The NYPD is responding to reports of multiple people shot inside Bronx-Lebanon Hospital at 1650 Grand Concourse, a police spokesman said.
According to an official who spoke to the AP news agency, the gunman was wearing a lab coat with a rifle concealed inside it.
Police were still trying to identify how many people had been shot.
The New York Times reported that three doctors were among those shot. The cops said that they are handling the situation and have gotten a report of a smoke condition on the 16th floor of the hospital.
Police had earlier asked city residents to avoid the area surrounding the hospital out of concern for public safety.
At least one doctor was treated inside the hospital by people who had tied an emergency fire hose as a tourniquet, a fire department official told the New York Times.
Bronx-Lebanon Hospital Center describes itself as the largest voluntary, not-for-profit health care system in the south and central Bronx. The hospital is around 120-year-old and claims that there are almost 1,000 beds spread across multiple units. It was not immediately clear whether police were looking for any other suspects.