How Res Net made very deep AI practical

Introduction

ResNet, short for Residual Network, became a landmark moment in artificial intelligence because it demonstrated that neural networks could be trained at depths that had previously seemed impractical. Before ResNet, researchers knew that deeper models should, in theory, learn richer representations. In practice, however, adding layers often made training harder and could even reduce accuracy. ResNet showed that this limitation was not an unavoidable consequence of depth itself. By redesigning how information moved through a network using shortcut paths, the architecture enabled models with more than 100 layers to train successfully and outperform much shallower systems. The results transformed deep learning research and helped establish a new belief: extreme depth could be useful if the architecture was designed correctly. [arXiv]arxiv.orgarXiv[1512.03385] Deep Residual Learning for Image RecognitionDecember 10, 2015 — by K He · 2015 · Cited by 320310 — On the ImageNet data…Published: December 10, 2015

What ResNet changed in deep network design

The importance of ResNet was not simply that it introduced another neural-network architecture. Its significance was that it provided convincing evidence against a widely held practical assumption of the time: that very deep networks were inherently too difficult to optimise.

The ResNet team focused on what they called the degradation problem. Researchers observed that when conventional networks became much deeper, training error could increase rather than decrease. This was surprising because a deeper network should always be capable of matching a shallower one by learning identity mappings in its extra layers. Yet standard optimisation methods often failed to find those solutions. [arXiv+2Wikipedia]arxiv.orgarXiv[1512.03385] Deep Residual Learning for Image RecognitionDecember 10, 2015 — by K He · 2015 · Cited by 320310 — On the ImageNet data…Published: December 10, 2015

ResNet addressed this by introducing residual blocks. Instead of forcing every layer stack to learn a completely new transformation, the architecture allowed information to travel along shortcut connections while the main path learned only the residual change. This made it easier for optimisation algorithms to preserve useful information and adjust it incrementally when necessary. The result was not merely a theoretical improvement; it changed the observed behaviour of deep networks during training. Models could become substantially deeper without suffering the severe optimisation difficulties that had previously appeared. [arXiv+2BibBase]arxiv.orgarXiv[1512.03385] Deep Residual Learning for Image RecognitionDecember 10, 2015 — by K He · 2015 · Cited by 320310 — On the ImageNet data…Published: December 10, 2015

A key reason the architecture proved so influential is that it reframed the question from “How can we train deeper networks?” to “How should information flow through deeper networks?” ResNet suggested that depth itself was not the enemy. The real challenge was ensuring that signals and gradients could move efficiently through many layers. [arXiv]arxiv.orgarXiv[1512.03385] Deep Residual Learning for Image RecognitionDecember 10, 2015 — by K He · 2015 · Cited by 320310 — On the ImageNet data…Published: December 10, 2015

Why ImageNet results made the idea convincing

Architectural ideas become influential when they produce measurable gains on important benchmarks. ResNet’s breakthrough came on ImageNet, the large-scale image-recognition challenge that served as one of the most important tests of computer vision systems in the 2010s.

The original ResNet paper reported successful training of a network with 152 layers, roughly eight times deeper than the influential VGG architecture that had previously set performance standards. Despite its extraordinary depth, the model achieved lower error rates while maintaining manageable computational complexity. [arXiv+2arXiv]arxiv.orgarXiv[1512.03385] Deep Residual Learning for Image RecognitionDecember 10, 2015 — by K He · 2015 · Cited by 320310 — On the ImageNet data…Published: December 10, 2015

Even more striking was the comparison with previous state-of-the-art systems. The 152-layer ResNet achieved a single-model top-5 validation error of 4.49%, outperforming results that had previously required ensembles of multiple models. An ensemble of ResNets later achieved a 3.57% error rate and won first place in the 2015 ImageNet Large Scale Visual Recognition Challenge (ILSVRC). [cv-foundation.org+2arXiv]cv-foundation.orgHe Deep Residual Learning CVPR 2016 paperThis single-model result outperforms all previous ensemble results (Table 5). We…

These results mattered because they showed that depth was delivering real performance gains rather than merely increasing model size. Researchers had long suspected that deeper networks could learn better representations, but ResNet provided a practical demonstration on a globally recognised benchmark. The architecture did not merely survive at extreme depth; it benefited from it. [cv-foundation.org]cv-foundation.orgHe Deep Residual Learning CVPR 2016 paperThis single-model result outperforms all previous ensemble results (Table 5). We…

The paper also presented experiments on CIFAR-10 with networks reaching hundreds and even one thousand layers. Although such extreme depths were not immediately practical for all applications, they reinforced the central message that optimisation barriers could be overcome. [arXiv]arxiv.orgarXiv[1512.03385] Deep Residual Learning for Image RecognitionDecember 10, 2015 — by K He · 2015 · Cited by 320310 — On the ImageNet data…Published: December 10, 2015

How ResNet changed thinking about depth

ResNet’s historical importance comes partly from what it disproved. Before its publication, many researchers viewed depth as a resource that quickly encountered diminishing returns. After ResNet, depth became a design dimension that could be expanded much more aggressively.

The architecture shifted the research community’s expectations. Rather than asking whether a network was too deep to train, researchers increasingly explored how architectural innovations could make even deeper systems feasible. Subsequent work investigated hundreds of layers, wider residual networks, densely connected networks, and other approaches that built directly on the idea that information should have efficient paths through a model. [Wikipedia]WikipediaResidual neural networkResidual neural network

This change in perspective was important for the broader development of AI. Many later advances depended on the assumption that optimisation challenges could be solved through architectural design. ResNet became one of the clearest demonstrations that seemingly fundamental limitations could sometimes be engineering problems rather than theoretical barriers. [BibBase]bibbase.orgWe explicitly reformulate…Read more…

How skip connections spread beyond computer vision

Although ResNet was created for image recognition, its deeper legacy lies in the spread of residual connections across AI.

Researchers quickly adopted residual-style shortcuts in object detection, semantic segmentation, and other computer-vision tasks. Variants such as ResNeXt, Wide ResNet, DenseNet, and Squeeze-and-Excitation Networks all built on or responded to the residual-learning framework introduced by ResNet. [Wikipedia]WikipediaResidual neural networkResidual neural network

The influence eventually extended far beyond vision. Residual connections became a common design pattern in modern deep learning because they improved optimisation and information flow. They now appear in many architectures that are not usually described as ResNets, including large language models and transformer-based systems. The specific implementations differ, but the underlying principle remains recognisable: preserve useful information through shortcut pathways while allowing layers to learn refinements rather than complete replacements. [Wikipedia]WikipediaResidual neural networkResidual neural network

In that sense, ResNet’s most enduring contribution was not a particular 152-layer network. It was the proof that extreme depth could work when information was allowed to move through a model in smarter ways. That insight reshaped neural-network design and remains embedded in many of the most powerful AI systems used today. [arXiv+2Wikipedia]arxiv.orgarXiv[1512.03385] Deep Residual Learning for Image RecognitionDecember 10, 2015 — by K He · 2015 · Cited by 320310 — On the ImageNet data…Published: December 10, 2015

Amazon book picks

Further Reading

Books and field guides related to How Res Net made very deep AI practical. Use these as the next step if you want deeper reading beyond the article.

Book

Deep Learning with Python

By Francois Chollet

Shows why architectural choices improve trainability and performance.

See on Amazon

Book

Dive into Deep Learning

By Aston Zhang, Zachary C. Lipton et al.

Explains modern architectures including residual-style concepts.

See on Amazon

Book

Neural Networks and Deep Learning

By Charu C. Aggarwal

Explains gradient propagation and training dynamics.

See on Amazon

Book

Deep Learning

By Ian Goodfellow, Yoshua Bengio et al.

Rating: 3.5/5 from 6 Google Books ratings

Covers optimisation challenges, gradient flow, and deep architectures.

See on Amazon

Browse more on Amazon: Deep Learning with Python Dive into Deep Learning Neural Networks and Deep Learning

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Shop location

Using USA USA USAUKCanadaAustraliaIreland

Example eBay listing

Anti AI Anti Machine Learning Say N Framed Wall Art Poster Canvas Print Picture

Search eBay.co.uk: machine learning wall art

Browse similar on eBay.co.uk

Example eBay listing

Machine Learning Framed Wall Art Poster Canvas Print Picture

Search eBay.co.uk: machine learning wall art

Browse similar on eBay.co.uk

Example eBay listing

Machine Learning Framed Wall Art Poster Canvas Print Picture

Search eBay.co.uk: machine learning wall art

Browse similar on eBay.co.uk

Browse more on eBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: arxiv.org
   Link: https://arxiv.org/abs/1512.03385
   Source snippet

   arXiv[1512.03385] Deep Residual Learning for Image RecognitionDecember 10, 2015 — by K He · 2015 · Cited by 320310 — On the ImageNet data...

   Published: December 10, 2015

2. Source: bibbase.org
   Link: https://bibbase.org/network/publication/he-zhang-ren-sun-deepresiduallearningforimagerecognition
   Source snippet

   We explicitly reformulate...Read more...

3. Source: Wikipedia
   Title: Residual neural network
   Link: https://en.wikipedia.org/wiki/Residual_neural_network

4. Source: arxiv.org
   Link: https://arxiv.org/abs/1512.03385?u=http%3A%2F%2Farxiv.org%2Fabs%2F1512.03385
   Source snippet

   [1512.03385] Deep Residual Learning for Image Recognitionby K He · 2015 · Cited by 321536 — On the ImageNet dataset we evaluate residual...

5. Source: cv-foundation.org
   Title: He Deep Residual Learning CVPR 2016 paper
   Link: https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf
   Source snippet

   This single-model result outperforms all previous ensemble results (Table 5). We...

6. Source: arxiv.org
   Link: https://arxiv.org/abs/2405.01725

7. Source: arxiv.org
   Link: https://arxiv.org/html/2405.01725v1
   Source snippet

   Development of Skip Connection in Deep Neural Networks...2 May 2024 — Previous studies have shown that the idea of skip connections or r...

   Published: May 2024

8. Source: arxiv.org
   Link: https://arxiv.org/pdf/1512.03385
   Source snippet

   1512.03385v1 [cs.CV] 10 Dec 2015by K He · 2015 · Cited by 320310 — ResNet reduces the top-1 error by 3.5% (Table 2), resulting from...

9. Source: resnet.us
   Link: https://www.resnet.us/about/us/
   Source snippet

   United States.Read more...

10. Source: mohitjain.me
    Link: https://mohitjain.me/2018/06/13/resnet/
    Source snippet

    The problem is the degradation problem; as we increase the depth of the network, instead of the accuracy increasing, it drops. This...Re...

11. Source: steggie3.github.io
    Link: https://steggie3.github.io/tech/resnet/
    Source snippet

    Deep Residual Learning for Image Recognition...19 Aug 2018 — ResNet is proposed in the 2015 paper Deep Residual Learning for Image Recog...

12. Source: rupeshks.cc
    Link: https://rupeshks.cc/blog/skip.html
    Source snippet

    The paper Identity Mappings in Deep Residual Networks has design mistakes leading to...Read more...

Additional References

1. Source: researchgate.net
   Link: https://www.researchgate.net/publication/311609041_Deep_Residual_Learning_for_Image_Recognition
   Source snippet

   Deep Residual Learning for Image RecognitionLater, ResNet addressed the degradation problem in deep neural networks through residual lear...

2. Source: deepseek.com
   Link: https://www.deepseek.com/en/
   Source snippet

   Free access to DeepSeek. Experience the intelligent model. Access API. Build with the latest DeepSeek models. Powerful models, sm...

3. Source: medium.com
   Link: https://medium.com/%40tnodecode/resnet-e7e0cba19e04
   Source snippet

   ResNet. How skip connections enabled very deep…How skip connections enabled very deep networks and tackled problems like vanishing gradie...

4. Source: viso.ai
   Link: https://viso.ai/deep-learning/resnet-residual-neural-network/
   Source snippet

   ResNet: Revolutionizing Deep Learning in Image RecognitionDeep residual networks like the popular ResNet-50 model are a convolutional neu...

5. Source: medium.com
   Link: https://medium.com/%40pranav.bhagwat_36/[understanding
   Source snippet

   Understanding the Residual Learning Framework for...The final 152-layer ResNet achieved a top 5 error rate of 3.57 percent on the ImageN...

6. Source: reddit.com
   Link: https://www.reddit.com/r/MachineLearning/comments/px3hzd/d_has_the_resnet_hypothesis_been_debunked/
   Source snippet

   [D] Has the ResNet Hypothesis been debunked?The ResNet architecture was [invented]({{ 'fake-citations-d81942/' | relative_url }}) to solve the degradation problem that has been empirical...

7. Source: scribd.com
   Link: https://www.scribd.com/document/677483373/ResNet
   Source snippet

   converge on the ImageNet dataset. ○ For too deep network, layers at later stages is. suspected to contribute...Read more...

8. Source: youtube.com
   Link: https://www.youtube.com/watch?v=C6tLw-rPQ2o
   Source snippet

   Deep Residual Learning for Image RecognitionWe have proposed deep residual networks which are very easy to train and they can simply gain...

9. Source: youtube.com
   Link: https://www.youtube.com/watch?v=cL8RqY1slI8
   Source snippet

   ResNet Explained — Deep Residual Learning for Image...A short walkthrough of "Deep Residual Learning for Image Recognition" by He, Zhang...

10. Source: sjpyo.tistory.com
    Title: Image Net Classification with Deep Convolutional Neural Networks;
    Link: https://sjpyo.tistory.com/72
    Source snippet

    residual learning for image recognition; ResNet (2015)- Try to resolve degradation problem with deeper network which isn't caused by over...