The urban poor experience serious discontents, harassment, eviction, police repression and local goons threatening when using urban space for living and livelihood purposes. This study pursues to understand the poor people’s negotiation strategies with different powerful agents who occupy money, muscleman and political affiliation. Following a mixed method approach, this study investigates the two biggest slums in Dhaka as case studies. Findings show that urban poor have to build different social-contract relations with various local agents as survival mechanisms while economic activities using urban space are considered to be illegal in Bangladesh. The role of the state is somewhat ambiguous in this regard. On the one hand, the state is not evicting the poor permanently from the city but it is repatriating them on other grounds and, on the other hand, it permits hundreds of informal intermediary agents to work for sustaining informal urban settlements for the poor people. We argue that these distinctive socio-structural arrangements in Dhaka city is hindering poor people from getting united and claiming their rights to the city while also not providing them proper opportunities to fully appropriate the urban space. These socio-economic relations need to be considered in order to make a just city for all, from the RTC perspective.