首页 技术 正文
技术 2022年11月14日
0 收藏 338 点赞 4,083 浏览 5512 个字

问题描述:

现网个别时候会出现CPU突然飙高的现象,飙高后不能恢复正常。

分析过程:

CPU飙高后抓dump,最好本机看,其它机器看dump可能需要下载服务运行机器的sos,clr

 
 

0:000> !threadpool

The version of SOS does not match the version of CLR you are debugging.  Please

load the matching version of SOS for the version of CLR you are debugging.

CLR Version: 4.0.30319.276

SOS Version: 4.0.30319.17929

CPU utilization: 100%

Worker Thread: Total: 54 Running: 54 Idle: 0 MaxLimit: 32767 MinLimit: 8

Work Request in Queue: 0

————————————–

Number of Timers: 3

————————————–

Completion Port Thread:Total: 10 Free: 10 MaxFree: 16 CurrentLimit: 8 MaxLimit: 1000 MinLimit: 8

CPU确实高,54个线程都在跑着。

接着看线程都在干什么

0:000> ~*e!clrstack

OS Thread Id: 0x3278 (57)

Child SP         IP               Call Site

000000000c72cea8 0000000077ef047a [RedirectedThreadFrame: 000000000c72cea8]

000000000c72cf40 000006448056c07e ********************.

SecurityQuota.ConcurrentLinkedQueue`1[[System.__Canon, mscorlib]].TryDequeue(System.__Canon ByRef)

000000000c72cff0 00000644805999e8 ********************.SecurityQuota.SecurityFeatureDigest.GetIncrementalDigest(********************)

……

生产环境服务CPU高问题分析

 
 

有50个线程在做该操作,review该处代码

为了防止本地代码和服务器不一致,直接反编译的服务器的代码,GetIncrementalDigest方法如下

public class SecurityFeatureDigest : ISecurityFeatureDigest

{

    // Fields

    private static Database _data = DatabaseManager.GetDatabase(“SQSDB”);

    private static readonly ITracing _tracer = TracingManager.GetTracing(typeof(SecurityFeatureDigest));

    private static ConcurrentLinkedQueue<List<FeatureDigest>> queue =

new ConcurrentLinkedQueue<List<FeatureDigest>>(new List<FeatureDigest>());     (2)

    private static readonly string[] SetFeatureDigestParameter = new string[] { “@digest”, “@operation”, “@sample”, “@stamp” };

 
 

    // 略…

 
 

    public void GetIncrementalDigest(RpcServerContext context)

    {

        DateTime args = context.GetArgs<DateTime>();

        List<FeatureDigest> results = new List<FeatureDigest>();

        List<FeatureDigest> result = null;

        while (true)

        {

            if (queue.TryDequeue(out result))      (1)

            {

                List<FeatureDigest> list3 = new List<FeatureDigest>();

                list3.AddRange(result);

                queue.Enqueue(result);

                foreach (FeatureDigest digest in list3)

                {

                    if (digest.Version > args)

                    {

                        digest.Sample = string.Empty;

                        results.Add(digest);

                    }

                }

                context.Return<List<FeatureDigest>>(results);

                return;

            }

        }

    }

 
 

    // 略…

 
 

}

从抓的dump看,极有可能是死循环了。通过以上代码看到,如果(1)处的返回为false的话,则改段代码会陷入死循环。接着看下什么情况下返回false。

public class ConcurrentLinkedQueue<T>

{

    // Fields

    private Node<T, T> _head;

    private Node<T, T> _tail;

 
 

    // 略…

   
 

    public bool TryDequeue(out T result)

    {

        while (true)

        {

            Node<T, T> comparand = this._head;

            Node<T, T> node2 = this._tail;

            Node<T, T> next = comparand.Next;

            if (comparand == this._head)

            {

                if (next == null)       (3)

                {

                    result = default(T);

                    return false;

                }

                if (comparand == node2)

                {

                    Interlocked.CompareExchange<Node<T, T>>(ref this._tail, next, node2);

                }

                else

                {

                    result = next.Item;

                    if (Interlocked.CompareExchange<Node<T, T>>(ref this._head, next, comparand) == comparand)

                    {

                        break;

                    }

                }

            }

        }

        return true;

    }

(3)处可以看出,如果head的next为null的话,则返回false。为了确认分析是否正确,看下此处的值是不是null

0:000> !dumpheap -stat

Statistics:

              MT    Count    TotalSize Class Name

000006448054b2d0        1           32 ********************.SecurityQuota.ConcurrentLinkedQueue`1

[[System.Collections.Generic.List`1[[********************.FeatureDigest, IICSecurityLibrary]], mscorlib]]

0:000> !dumpheap – -mt 000006448054b2d0   

——————————

Heap 6

         Address               MT     Size

00000001404836f0 000006448054b2d0       32    

total 0 objects

Statistics:

              MT    Count    TotalSize Class Name

           32 ********************.SecurityQuota.ConcurrentLinkedQueue`1

[[System.Collections.Generic.List`1[[********************.FeatureDigest, *****]], mscorlib]]

Total 1 objects

0:000> !do 00000001404836f0

Name:        ********************.SecurityQuota.ConcurrentLinkedQueue`1[[System.Collections.Generic.List`1

[[********************.FeatureDigest, *****]], mscorlib]]

MethodTable: 000006448054b2d0

EEClass:     0000064480557a90

Size:        32(0x20) bytes

File:        ********************\SecurityQuotaService.exe

Fields:

              MT    Field   Offset                 Type VT     Attr            Value Name

        8 …Canon, mscorlib]]  0 instance 00000000c3135ec8 _head

       10 …Canon, mscorlib]]  0 instance 00000000c3135ec8 _tail

0:000> !do 00000000c3135ec8

Name:        ********************.SecurityQuota.ConcurrentLinkedQueue`1+Node`1[[System.Collections.Generic.List`1[[********************.

FeatureDigest, *****]], mscorlib],[System.Collections.Generic.List`1[[********************.FeatureDigest, *****]], mscorlib]]

MethodTable: 000006448054b7f8

EEClass:     00000644805580b8

Size:        32(0x20) bytes

File:        ********************\SecurityQuotaService.exe

Fields:

              MT    Field   Offset                 Type VT     Attr            Value Name

        8       System.__Canon  0 instance 00000001404836a8 Item

       10 …Canon, mscorlib]]  0 instance 0000000000000000 Next

发现Next为null

 
 

问题结论:

由于一些异常逻辑导致程序死循环,此处代码也有问题,对异常逻辑处理不够。

 
 

 

作者:No.40

Blog:http://www.cnblogs.com/no40

 

相关推荐
python开发_常用的python模块及安装方法
adodb:我们领导推荐的数据库连接组件bsddb3:BerkeleyDB的连接组件Cheetah-1.0:我比较喜欢这个版本的cheeta…
日期:2022-11-24 点赞:878 阅读:9,497
Educational Codeforces Round 11 C. Hard Process 二分
C. Hard Process题目连接:http://www.codeforces.com/contest/660/problem/CDes…
日期:2022-11-24 点赞:807 阅读:5,910
下载Ubuntn 17.04 内核源代码
zengkefu@server1:/usr/src$ uname -aLinux server1 4.10.0-19-generic #21…
日期:2022-11-24 点赞:569 阅读:6,744
可用Active Desktop Calendar V7.86 注册码序列号
可用Active Desktop Calendar V7.86 注册码序列号Name: www.greendown.cn Code: &nb…
日期:2022-11-24 点赞:733 阅读:6,497
Android调用系统相机、自定义相机、处理大图片
Android调用系统相机和自定义相机实例本博文主要是介绍了android上使用相机进行拍照并显示的两种方式,并且由于涉及到要把拍到的照片显…
日期:2022-11-24 点赞:512 阅读:8,135
Struts的使用
一、Struts2的获取  Struts的官方网站为:http://struts.apache.org/  下载完Struts2的jar包,…
日期:2022-11-24 点赞:671 阅读:5,298