• brucethemoose@lemmy.world
    link
    fedilink
    English
    arrow-up
    15
    arrow-down
    1
    ·
    edit-2
    2 days ago

    Qwen 2.5 is already amazing for a 14B, so I don’t see how deepseek can improve that much with a new base model, even if they continue train it.

    Perhaps we need to meet in the middle, and have quad channel APUs like Strix Halo become more common, and maybe release like 40-80GB MoE models. Perhaps bitnet ones?

    Or design them for asynchronous inference.

    I just don’t see how 20B-ish models can perform like one orders of magnitude bigger without a paradigm shift.

    • zecg@lemmy.world
      link
      fedilink
      English
      arrow-up
      5
      ·
      1 day ago

      I use 14b and it’s certainly great for my modest highschool physics and python (to help the kids) needs, but for party games and such it’s a drag its pop culture stops at mid 2023

      • brucethemoose@lemmy.world
        link
        fedilink
        English
        arrow-up
        2
        ·
        22 hours ago

        Thing is, there are a lot of free APIs for 30B-70B class models.

        Self hosting is great of course, and if 14B does the job then it does the job.

    • cyd@lemmy.world
      link
      fedilink
      English
      arrow-up
      6
      ·
      1 day ago

      Intriguingly, there’s reason to believe the R1 distills are nowhere close to their peak performance. In the R1 paper they say that the models are released as proofs of concept of the power of distillation, and the performance can probably be improved by doing an additional reinforcement learning step (like what was done to turn V3 into R1). But they said they basically couldn’t be bothered to do it and are leaving it for the community to try.

      2025 is going to be very interesting in this space.

    • Teanut@lemmy.world
      link
      fedilink
      English
      arrow-up
      7
      ·
      2 days ago

      nVidia’s new Digits workstation, while expensive from a consumer standpoint, should be a great tool for local inferencing research. $3000 for 128GB isn’t a crazy amount for a university or other researcher to spend, especially when you look at the price of the 5090.

      • Viri4thus@feddit.org
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 day ago

        Why would you buy a single use behemoth when you can buy a strix halo 128GB that can work as an actual tablet/laptop and have all the functionality of the behemoth?! while supporting decades of legacy x86 software. Truly wondering why anyone would buy that NVIDIA thing other than pure ignorance and marketing says NV is the AI company.

      • brucethemoose@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 day ago

        Dense models that would fit in 100-ish GB like mistral large would be really slow on that box, and there isn’t a SOTA MoE for that size yet.

        So, unless you need tons of batching/parallel requests, its… kinda neither here nor there?

        As someone else said, the calculus changes with cheaper Strix Halo boxes (assuming those mini PCs are under $3K).