I am not into 'gritty urbanism'. Faded paint, pealing plaster, cracked cement - buildings that look sad and tired. Seas of gray and the overuse of cement. Mid-century 'modern' architecture. Massively wide and barren streets. There is a lot of gritty urbanism. Plenty of people look upon these places and see opportunity and embrace it. I think many of these efforts come from folks in predominantly suburban areas that tolerate the grit to preserve and eventually improve what little urbia remains. I see no problem with that, and for some people, that may be their life's calling. These people are important, because the United States has plenty of cheap gritty urbia, and wanting to nurture and improve your environment is a noble cause. But, I think rugged urban pioneers are somewhat of an exception, and most people - with a full time job to go to during the day, and likely a family at home to raise - will try to live in as nice of an area that they can reasonably afford.
When you relocate to a new city, one of the first things you have to do is find a place to live. When I relocated to the New York City area with my wife, we considered several different neighbourhoods. Neither of us are in to gritty urbanism, and we quickly found that there is a real shortage of quality urban neighbourhoods in the NYC area, and that narrowed our search down. The further we narrowed our search, the higher the prices went, and eventually we chose the nicest home that we could reasonably afford, and we found ourselves in Hoboken, NJ, a city which many have described as being 'gentrified'.
Real estate prices are mostly the cause supply and demand (an exception to this is new development in extremely cheap areas where housing can be sold close to construction costs, but this does not apply to established housing stock.) Let's say there is a supply of 1,000 housing units in a neighbourhood, but a demand of 100,000 people to live there. Those people are going to compete against each other and prices are going to rise to the point that only the top 1,000 of those 100,000 (or 1%) can afford to live there. New York City is home to about 8.5 million people (or 20 million in the metropolitan area) and has a diverse mixture of billionaires, poor, and everything in between. If there is a theoretical neighbourhood where everyone wants to live, but it can only accommodate 1% of New York City's population, it will only be affordable to the top 1% of New York City's residents, and rents are going to be extremely high. It should be no surprise that Greenwich Village, one of the nicest areas in NYC but can only house 0.2% of New York City's population, can get away with selling multi-million dollar one bedroom homes.
There is a huge lack of supply of quality urban neighbourhoods in the United States. As Americans are slowly moving back into cities, and companies are choosing urban locations over suburban locations, we are adding to the demand to live in quality urban places without increasing the supply so naturally prices are going to increase. Especially in cities such as New York City or San Francisco which already have extremely high rent (indicating a shortage of housing supply in general), it should also not surprise us that if we improve an area so that it becomes nicer than the typical neighbourhood in the city, there will be a huge demand to live there and real estate prices will skyrocket.
Gentrification has a bad reputation. Gentrification, in its most extreme form, describes a process where initially cheap housing attracts residents, who clean up and improve the neighbourhood, only for the neighbourhood now to be attractive to wealthier residents, which attracts better services and amenities, which in term attracts even wealthier residents. Effectively, the nicer you make an area, the more people that want to live there. If the demand to live there grows faster than the supply of housing, it will cost more. This stirs up a kerfuffle, because the former residents can no longer afford to live there, and that is why gentrification has a bad reputation. It is not that dramatic in reality, but the effect is obvious. The non-gritty parts of New York City and San Francisco have rents that are often more than the average American's income. Everyone wants to live in a nice area, and it should not be unreasonably expensive to do so.
The solution to housing affordability is not government intervention such as rent control and zoning, because if you limit both the supply and price of housing the result is that you simply have no housing; then you might end up in a similar situation as Sweden where you go on a waiting list for many years just to get a home. The solution is not to avoid improving your neighbourhood to prevent any demand to live there. The solution is not to build more cheap over-supplied suburban sprawl; that is like thinking that we can somehow lower the price of sneakers by supplying more boots. We might induce demand for boots, but it is not going to satisfy the demand for more sneakers.
The solution to housing affordability (and dampening the negative effects of gentrification) is to build more quality urban neighbourhoods where people want to actually live, and allowing the market to supply the types of housing that people want in the areas that they want.