Rocky Sullivan (James Cagney) and Jerry Connolly (Pat O'Brien) are childhood friends who robbed a railroad car as kids. Rocky saved Jerry's life during the chase by pulling him out of the way of a steam train while running from the guards that saw them. Rocky was then caught by the police, but Jerry - who could run faster - escaped. Rocky, after being sent to reform school, grows up to become a notorious gangster, while Jerry has become a priest.
Rocky returns to his old neighborhood, where Jerry is running a home that intends to keep young boys away from a life of crime. Six of those boys, Soapy (Billy Halop), Swing (Bobby Jordan), Bim (Leo Gorcey), Patsy (Gabriel Dell), Crabface (Huntz Hall), and Hunky (Bernard Punsly), idolize Rocky, and Jerry attempts to keep his former friend from corrupting them. (These boys were to star in Dead End Kids/East Side Kids/The Bowery Boys films).
Meanwhile Rocky gets involved with Frazier (Humphrey Bogart), a crooked lawyer, and Keefer (George Bancroft), a shady businessman and municipal contractor. They try to dispose of Rocky, but he finds the record book that they keep where they list the bribes to city officials. Jerry learns of these events and warns Rocky to leave before he informs the authorities. Rocky ignores his advice and Jerry gets the public's attention and informs them all of the crooked government, causing Frazier and Keefer to plot to kill him. Rocky overhears this plot and kills them to protect his childhood friend.
Rocky is then captured following an elaborate shootout in a building, and sentenced to die. Jerry visits him just before his execution and asks him to do him one last favor - to die pretending to be a screaming, snivelling coward, which would end the boys' idolization of him. Rocky refuses, and insists he will be 'tough' to the end, and not give up the one thing he has left, his pride. At the very last moment he appears to change his mind and has to be dragged to the elect