Sep 19,Notes on Qwen2.5 | Singularity Gallery

type

status

date

slug

summary

Introduction

The Qwen Team has released the new Qwen2.5 series models, potentially the largest open-source release in history. This release features the Qwen2.5 language models (LLMs) along with specialized expert models for coding (Qwen2.5-Coder) and mathematics (Qwen2.5-Math). These models are available in various sizes:

Qwen2.5: 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B

Qwen2.5-Coder: 1.5B, 7B, and 32B (coming soon)

Qwen2.5-Math: 1.5B, 7B, and 72B

All models, except for the 3B and 72B variants, are licensed under Apache 2.0. Additionally, the Qwen Team offers APIs for their flagship language models—Qwen2.5-Plus and Qwen2.5-Turbo—through Model Studio. They've also released Qwen2-VL-72B, featuring performance enhancements compared to last month's release.

Details of the LLM

The Qwen2.5-LLM series represents a significant advancement in language model capabilities, offering a range of decoder-only dense models from 0.5B to 72B parameters. These models cater to user interests, particularly in the 10B-30B range for production, with open-sourced models including Qwen2.5-3B, 14B, and 32B.

Key features of the series include: • Larger pre-training dataset • Enhanced knowledge base • Improved coding and mathematical abilities • Better alignment with human preferences • Enhanced performance in instruction following, long-text generation, and structured data processing

Performance metrics demonstrate the Qwen2.5-LLM models' superiority over competitors in benchmark evaluations across general tasks, mathematics, coding, and multilingualism. The Qwen2.5-72B model excels despite using fewer parameters than some rivals.

Instruction-tuned models further enhance capabilities in natural language understanding, question answering, reasoning, and alignment tasks. The Qwen2.5-72B-Instruct model shows exceptional results in mathematics, coding, and chatting.

The series also includes edge-side models like Qwen2.5-0.5B and 1.5B, which maintain strong performance for resource-constrained environments.

Following is the model card of Qwen2.5-LLM:

Qwen2.5-72B Performance:

Qwen2.5-14B/32B Performance:

For more details on model evaluation, visit https://qwenlm.github.io/blog/qwen2.5-llm/.

Details of Qwen2.5-Coder

QwenLM's latest release, Qwen2.5-Coder, marks a significant leap in their coding model series. Now available in sizes up to 32B, it's designed to be a comprehensive coding partner. This iteration scales up code training data to a staggering 5.5 trillion tokens, enhancing coding capabilities while maintaining the model's strength in math and general tasks.

The model supports an extensive context of up to 128K tokens and covers 92 programming languages. It shows remarkable improvements in code-related tasks, even outperforming larger models in certain evaluations. The instruction-tuned variant, Qwen2.5-Coder-Instruct, further enhances task performance and generalization. It excels in multi-programming, code reasoning, and math reasoning, while retaining the general abilities of the base Qwen2.5 model.

Qwen2.5-Coder is released under the Apache 2.0 license, with plans to release a 32B version soon.

Qwen2.5-Coder-Instruct Performance:

You can see more details at https://qwenlm.github.io/blog/qwen2.5-coder/

Details of Qwen2.5-Math

Qwen2.5-Math significantly upgrades its predecessor, offering a suite of models: base (1.5B/7B/72B), instruction-tuned (1.5B/7B/72B-Instruct), and a mathematical reward model (RM-72B). These models excel in tackling mathematical challenges with enhanced precision and reasoning, particularly in computational accuracy and complex problem-solving.

Rigorously evaluated against comprehensive English and Chinese math benchmarks, the Qwen2.5-Math series demonstrates notable performance gains over its predecessor. The integration of TIR has boosted the models' proficiency in precise computation, symbolic manipulation, and algorithmic reasoning.

The iterative development process includes: • Synthesizing high-quality mathematical pre-training data • Aggregating additional mathematical data • Leveraging the Qwen2.5 series base model for enhanced language understanding and reasoning

The Qwen2.5-Math-Instruct models have been further refined using a math-specific reward model and reinforcement learning, leading to superior performance in both English and Chinese mathematical benchmarks. The flagship Qwen2.5-Math-72B-Instruct model sets a new standard for open-source mathematical models, outperforming both open-source and leading closed-source competitors.

Qwen2.5-Math Performance:

For more details, please check the https://qwenlm.github.io/blog/qwen2.5-math/

Test Cases of Qwen2.5-LLM

I test the Qwen2.5-LLM using their demo deployed on Huggingface. Here is the link you can try: https://huggingface.co/spaces/Qwen/Qwen2.5

Firstly, I try some classical tricky test questions.

Question 1: If Alice has 5 sisters and 6 brothers, how many sisters does her brother have?

Qwen2-72B:

🔑

Alice has 5 sisters and 6 brothers. Since all of Alice's siblings share the same set of siblings, including Alice herself, we can determine the number of sisters her brother has as follows:

Alice's brother has the same number of sisters as Alice does.

Alice has 5 sisters (including herself).

Therefore, Alice's brother has 5 sisters.

Question 2: What is the smallest integer whose square is between 15 and 30? Please consider the negative

Qwen2.5-72B:

🔑

To find the smallest integer whose square is between 15 and 30, we need to identify the integers nnn such that:

15<n2<3015 < n^2 < 3015<n2<30

First, let's find the positive integers that satisfy this condition.

Calculate the square root of 15:

15≈3.87\sqrt{15} \approx 3.8715≈3.87

So, nnn must be greater than 3.87.

Calculate the square root of 30:

30≈5.48\sqrt{30} \approx 5.4830≈5.48

So, nnn must be less than 5.48.

The positive integers that satisfy 3.87<n<5.483.87 < n < 5.483.87<n<5.48 are 4 and 5.

Next, let's check the squares of these integers:

42=164^2 = 1642=16

52=255^2 = 2552=25

Both 16 and 25 are between 15 and 30.

Now, let's consider the negative integers. The negative integers that satisfy 15<n2<3015 < n^2 < 3015<n2<30 will be the negatives of the positive integers we found.

(−4)2=16(-4)^2 = 16(−4)2=16

(−5)2=25(-5)^2 = 25(−5)2=25

So, the negative integers that satisfy the condition are -4 and -5.

Among the integers 4, 5, -4, and -5, the smallest integer is -5.

Therefore, the smallest integer whose square is between 15 and 30 is −5.

Question3:Which one is bigger? 13.11 or 13.8?

Qwen2.5-72B:

🔑

To determine which number is bigger between 13.11 and 13.8, you can compare them digit by digit from left to right:

Compare the whole number parts:

Both numbers have the same whole number part, which is 13.

Compare the decimal parts:

The first decimal place: 1 (from 13.11) and 8 (from 13.8).

Since 8 is greater than 1, 13.8 is greater than 13.11.

Therefore, 13.8 is bigger than 13.11.

Question4:Please exchange the second word and the last second word for the following sentence:I need to go to work even though today is Saturday.

Qwen2.5-72B

🔑

Sure, here is the sentence with the second word and the second-to-last word exchanged:

I Saturday to go to work even though today is need.

Question5:If it takes 1 hour to dry 25 clothes under the sun, how long will it take to dry 30 clothes?

Qwen2.5-72B

🔑

The time it takes to dry clothes under the sun generally depends on factors like sunlight intensity, humidity, and wind, rather than the number of clothes, assuming they all have enough space and exposure to the same conditions. If 25 clothes take 1 hour to dry, then 30 clothes would also take approximately 1 hour to dry, provided they are spread out in a similar manner and have similar exposure to the sun.

However, if the drying area is limited and the 30 clothes are more crowded or overlapping, it might take longer due to reduced air circulation and sunlight exposure. But based on the information given, we can assume that the drying time remains the same:

It will take about 1 hour to dry 30 clothes.

Question6:what is the fifth word in the following sentences?I have an apple

Qwen2.5-72B

🔑

The fifth word in the sentence "I have an apple" is "apple". However, it's worth noting that this sentence only contains four words. If you intended to ask about a different sentence or if there was a specific context or longer sentence you had in mind, please provide more details!

Some Insights

I believe Qwen2.5-72B is a remarkably powerful model. As an open-source offering, it correctly answered 4 out of 6 questions. I've tested these questions on various models including Gemini, o1-preview, and deepseek-v2.5, among others. Interestingly, no model could solve all these problems. For a 70B parameter model, Qwen2.5 demonstrates significantly stronger reasoning capabilities compared to its predecessor, Qwen2.

<ins/>