日期:2014-05-18  浏览次数:20965 次

C#读文件存储到数据库,求思路
有一个txt文件 格式如下
111117105000000001,111117105000000001=99122012570000,686501,0000001
111117105000000002,111117105000000002=99122013110000,118653,0000002
111117105000000003,111117105000000003=99122015150000,484460,0000003
111117105000000004,111117105000000004=99122011310000,437451,0000004
111117105000000005,111117105000000005=99122018500000,627604,0000005
111117105000000006,111117105000000006=99122012430000,029250,0000006
111117105000000007,111117105000000007=99122016070000,334081,0000007
111117105000000008,111117105000000008=99122013850000,564231,0000008
111117105000000009,111117105000000009=99122017340000,751953,0000009
111117105000000010,111117105000000010=99122017420000,108740,0000010
文件最小为6M多,10万行数据
我是这样操作的
C# code

string[] temp = System.IO.File.ReadAllLines(SavePath, System.Text.Encoding.GetEncoding("gb2312"));
IDictionary<bool, ArrayList> dict = BusiNeobyCardImportBiz.ImportCard(temp);
public IDictionary<bool, ArrayList> ImportCard(string[] ListCard)
        {
            int Success = 0;//成功次数
            int Failure = 0;//失败次数
            IDictionary<bool, ArrayList> results = new Dictionary<bool, ArrayList>();
            ArrayList arraylist = new ArrayList();
            for (int i = 0; i < ListCard.Length; i++)
            {
                if (string.IsNullOrEmpty(ListCard[i]))
                {
                    continue;
                }
                string[] cardNote = ListCard[i].Split(',');
                if (cardNote.Length != 4||cardNote[0].Length!=18)
                {
                    Failure++;
                    arraylist.Add(ListCard[i]);
                    continue;
                }
                if (this.Find(ExpEnter.Equal("CardNo", cardNote[0]), null).Count >= 1)
                {
                    Success++;
                    continue;
                }
                BusiNeobyCardImport BusiNeobyCardImportModel = new BusiNeobyCardImport();
                BusiNeobyCardImportModel.CardNo = cardNote[0];
                BusiNeobyCardImportModel.Note1 = cardNote[0];
                BusiNeobyCardImportModel.Note2 = cardNote[2];
                BusiNeobyCardImportModel.Note3 = cardNote[3];
                BusiNeobyCardImportModel.Status = "N";
                OpResult result = BusiNeobyCardImportDal.Add(BusiNeobyCardImportModel);//添加到数据库
                if (result.Success)
                {
                    Success++;
                }
                else
                {
                    Failure++;
                    arraylist.Add(ListCard[i]);
                }
            }
            ArrayList listsuccess = new ArrayList();
            listsuccess.Add(Success);
            results.Add(false, arraylist);
            results.Add(true, listsuccess);
            return results;
        }


如果文件过大,总不能全部放到内存处理。
我想逐行读取,然后操作。
想请问下有没有更好的思路

------解决方案--------------------
既然你要写入数据库,那么为何不直接读取上来存到DataTable中?6M的文件全部读到内存应该也没什么问题,200M以下都不会有太大问题,超过200M可以考虑逐行读取,

------解决方案--------------------
用读文件类一次只读一行.处理完后,存入数据库中.看看System.IO.File名称空间下的FIleRead类.好像是.有点记不得了
------解决方案--------------------
探讨

引用:
既然你要写入数据库,那么为何不直接读取上来存到DataTable中?6M的文件全部读到内存应该也没什么问题,200M以下都不会有太大问题,超过200M可以考虑逐行读取,

存到DataTable用SqlBulkCopy实现速度应该会快很多
我正打算用SqlBulkCopy试下要多少时间。

------解决方案--------------------
如果实在觉得太大的话,可以分开一批一批的读取插入行不?
------解决方案--------------------
1W行,1秒中
------解决方案--------------------