Core-to-Core Latency

As the core count of modern CPUs is growing, we are reaching a time when the time to access each core from a different core is no longer a constant. Even before the advent of heterogeneous SoC designs, processors built on large rings or meshes can have different latencies to access the nearest core compared to the furthest core. This rings true especially in multi-socket server environments.

But modern CPUs, even desktop and consumer CPUs, can have variable access latency to get to another core. For example, in the first generation Threadripper CPUs, we had four chips on the package, each with 8 threads, and each with a different core-to-core latency depending on if it was on-die or off-die. This gets more complex with products like Lakefield, which has two different communication buses depending on which core is talking to which.

If you are a regular reader of AnandTech’s CPU reviews, you will recognize our Core-to-Core latency test. It’s a great way to show exactly how groups of cores are laid out on the silicon. This is a custom in-house test, and we know there are competing tests out there, but we feel ours is the most accurate to how quick an access between two cores can happen.


(Click on the image to enlarge)

Analyzing core-to-core latencies on the AMD Ryzen Threadripper 7980X (64C/128T), our test is limited to probing the first 64 threads, although scaling out to 128 threads would be identical. Each CCD on the Threadripper 7980X has 8 x Zen 4 cores, with 32 MB of L3 cache. Looking at the latency range within the CCD, we can see inner latencies between 7 and 20 ns, which increase to 89 and 96 ns as each core communicates within the CCX.


A visual render of the AMD Ryzen Threadripper 7980X with 8 x CCDs and IOD

Given we've reviewed the AMD Ryzen 9 7950X, which has the same Zen 4 cores and the same CCD complex approach to communicating between the cores, we see relatively similar latencies in both Threadripper 7000 and Ryzen 7000. A quad-channel DDR5 memory controller integrated within the large IOD and using PCIe 5.0 lanes as the primary pathway is important in enhancing the Infinity Fabric interconnect to reduce latencies and help counteract any penalties.

SPEC2017 Multi-Threaded Results Threadripper 7000 vs. Threadripper 3000: Generational Improvements
Comments Locked

66 Comments

View All Comments

  • Makaveli - Monday, November 20, 2023 - link

    So how did you not catch that the memory you were looking at was using Hynix?

    I just had to look at the specs and I knew just by the Cas latency. There was no good memory in the DDR4 range that came in at CL18.
  • meacupla - Tuesday, November 21, 2023 - link

    Well, clearly, the site that I used thought it was B-die, when it was false info for the 2x16GB model.
  • tamalero - Tuesday, December 5, 2023 - link

    I remember when samsung released low quality "b" dies. It was in the news i think in tomshardware.
    These were used as "b dies" in corsair high end ram.
    And they were not as good as the top class high binned true b die.
    Someone correct me If I am wrong.
  • 29a - Tuesday, November 21, 2023 - link

    The PCB the memory is mounted on matters too.
  • kn00tcn - Tuesday, November 21, 2023 - link

    micron m-die(?) 3600c16 working great on am4, it just wasnt available until a couple years after launch, samsung isnt the only choice

    and actually hynix had different dies, some tighter and more stable than others

    first hand anecdotal:

    1) in 2018 hynix cjr(?) 3200c16 / 2600x / msi b450m mortar = never fully stable, had to tweak low level ohms, maybe the cpu had issue, linux randomly showed amdgpu pcie timeouts in log

    2) though in 2023 the same hynix sticks work fine with 5600g / asrock deskmeet

    3) in 2020 micron 3600c16 / 3600x / asus tuf b450m = solid

    4) in 2023 different micron 3600c16 / 5600 with pbo / same msi b450m mortar from 2018, everything fine
  • demu - Tuesday, November 21, 2023 - link

    Before b-die memory I also had a G.Skill Trident set with Hynix ic:s (G.Skill 64GB (4 x 16GB) Trident Z, DDR4 3600MHz, CL17, 1.35V CL17-19-19-19).
    They also worked @3600 CL16-18-18-18 or 3733 CL17.
  • iamkyle - Monday, November 20, 2023 - link

    I see unlike previous generations of Threadripper, AMD and its board partners are abandoning the "content creator/gamer" segment.

    Great for the workstation crowd, a loss for the aforementioned.
  • Threska - Monday, November 20, 2023 - link

    The people who created Crysis could have used this. :-)
  • kn00tcn - Tuesday, November 21, 2023 - link

    and what does a content creator need tons of pcie lanes and quad+ channel memory for? regular desktop isnt weak with 16 high freq cores and 3d cache

    if workloads like rendering scale so well with cores then they also scale across multiple networked computers for a cost effective render farm instead of a single expensive threadripper

    main issue is probably the daw niche of extremely complex realtime audio synths/effects, but this has workarounds for years (prerender specific tracks), and it's not like we had better performance available in the past
  • thestryker - Monday, November 20, 2023 - link

    HEDT isn't back when the platform price of entry is more than double that of the top desktop setups. AMD did release Zen 4 TR cheaper than Intel's current closest equivalent ($1500 vs $2100), but when $1500 is the cheapest it gets CPU wise you could put together CPU/DRAM/mobo for less. This is why I've contended TR 3xxx actually marked the end of HEDT as that is when the price of entry became significantly higher than desktop.

Log in

Don't have an account? Sign up now