本篇文章给大家谈谈SqlBulkCopyPrecisionErrorfromC#DoubletoSqlDecimal?,同时本文还将给你拓展c#–如何使用SqlBulkCopyColumnMappin
本篇文章给大家谈谈SqlBulkCopy Precision Error from C# Double to Sql Decimal?,同时本文还将给你拓展c# – 如何使用SqlBulkCopyColumnMappingCollection?、c# – 让SqlBulkCopy在MiniProfiler中显示为sql、Code Sign error: No matching provisioning profile found: Your build settings specify a provisioni...、EntityFramework Core 5.0 VS SQLBulkCopy等相关知识,希望对各位有所帮助,不要忘了收藏本站喔。
本文目录一览:- SqlBulkCopy Precision Error from C# Double to Sql Decimal?
- c# – 如何使用SqlBulkCopyColumnMappingCollection?
- c# – 让SqlBulkCopy在MiniProfiler中显示为sql
- Code Sign error: No matching provisioning profile found: Your build settings specify a provisioni...
- EntityFramework Core 5.0 VS SQLBulkCopy
SqlBulkCopy Precision Error from C# Double to Sql Decimal?
https://forums.asp.net/t/1300113.aspx?SqlBulkCopy+Precision+Error+from+C+Double+to+Sql+Decimal+
Before I bulkcopy data into the database, I use DataTable to store the bulk of data.
Since the value to be stored is declared as double in the DataTable, I defined the datatype of that data column to be double too.
This causes a conversion problem as somewhere along the chain (I am suspecting the SqlBulkCopy is doing some kind of implicit conversion) the precision of my value was changed.
Ever since I changed the data type of that data column from double into decimal and explicitly convert the datatype before storing into the DataTable, the problem is fixed.
C# double to sql server lost precision
private void FixDoubleToDecimal(DataSet dataSet)
{
if (dataSet == null)
{
return;
}
foreach (DataTable table in dataSet.Tables)
{
foreach (DataColumn column in table.Columns)
{
if (column.DataType == typeof(double))
{
column.DataType = typeof(decimal);
}
}
}
}
测试代码
[Test]
public void DecimalTest()
{
DataTable dataTable=new DataTable();
DataColumn dataColumn=new DataColumn("column1"){DataType = typeof(double)};
dataTable.Columns.Add(dataColumn);
DataRow dataRow= dataTable.NewRow();
dataRow["column1"] = 0.000002552;
Console.WriteLine(dataRow["column1"]);
dataTable.Columns["column1"].DataType = typeof(decimal);
Console.WriteLine(dataRow["column1"]);
}
有数据就无法转换
System.ArgumentException : Cannot change DataType of a column once it has data.
at System.Data.DataColumn.set_DataType(Type value)
at AssemblyTest.MainTest.DecimalTest() in C:\Users\clu\source\repos\Edenred\Test\AssemblyTest\MainTest.cs:line 492.552E-06
How To Change DataType of a DataColumn in a DataTable?
方案1
Old post, but I thought I''d weigh in, with a DataTable extension that can convert a single column at a time, to a given type:
public static class DataTableExt
{
public static void ConvertColumnType(this DataTable dt, string columnName, Type newType)
{
using (DataColumn dc = new DataColumn(columnName + "_new", newType))
{
// Add the new column which has the new type, and move it to the ordinal of the old column
int ordinal = dt.Columns[columnName].Ordinal;
dt.Columns.Add(dc);
dc.SetOrdinal(ordinal);
// Get and convert the values of the old column, and insert them into the new
foreach (DataRow dr in dt.Rows)
dr[dc.ColumnName] = Convert.ChangeType(dr[columnName], newType);
// Remove the old column
dt.Columns.Remove(columnName);
// Give the new column the old column''s name
dc.ColumnName = columnName;
}
}
}
It can then be called like this:
MyTable.ConvertColumnType("MyColumnName", typeof(int));
Of course using whatever type you desire, as long as each value in the column can actually be converted to the new type.
DataColumn.Ordinal Gets the (zero-based) position of the column in the DataColumnCollection collection.
评论
问 Gives error "Object must implement IConvertible." while converting Byte[] to string type column.
public static void ConvertColumnType<T>(this DataTable dt, string columnName, TnewType) where T : Type, IConvertible
–
dr[dc.ColumnName] = dr[columnName] == DBNull.Value ? DBNull.Value : Convert.ChangeType(dr[columnName], newType);
方案2
I created an extension function which allows changing the column type of a DataTable. Instead of cloning the entire table and importing all the data it just clones the column, parses the value and then deletes the original.
/// <summary>
/// Changes the datatype of a column. More specifically it creates a new one and transfers the data to it
/// </summary>
/// <param name="column">The source column</param>
/// <param name="type">The target type</param>
/// <param name="parser">A lambda function for converting the value</param>
public static void ChangeType(this DataColumn column, Type type, Func<object, object> parser)
{
//no table? just switch the type
if (column.Table == null)
{
column.DataType = type;
return;
}
//clone our table
DataTable clonedtable = column.Table.Clone();
//get our cloned column
DataColumn clonedcolumn = clonedtable.Columns[column.ColumnName];
//remove from our cloned table
clonedtable.Columns.Remove(clonedcolumn);
//change the data type
clonedcolumn.DataType = type;
//change our name
clonedcolumn.ColumnName = Guid.NewGuid().ToString();
//add our cloned column
column.Table.Columns.Add(clonedcolumn);
//interpret our rows
foreach (DataRow drRow in column.Table.Rows)
{
drRow[clonedcolumn] = parser(drRow[column]);
}
//remove our original column
column.Table.Columns.Remove(column);
//change our name
clonedcolumn.ColumnName = column.ColumnName;
}
}
You can use it like so:
List<DataColumn> lsColumns = dtData.Columns
.Cast<DataColumn>()
.Where(i => i.DataType == typeof(decimal))
.ToList()
//loop through each of our decimal columns
foreach(DataColumn column in lsColumns)
{
//change to double
column.ChangeType(typeof(double),(value) =>
{
double output = 0;
double.TryParse(value.ToString(), out output);
return output;
});
}
The above code changes all the decimal columns to doubles.
最终的解决方案
DataSet data;
data.ConvertColumnType(typeof(double), typeof(decimal));
public static void ConvertColumnType(this DataTable dt, string columnName, Type newType)
{
using (DataColumn dc = new DataColumn($"{columnName}_new", newType))
{
// Add the new column which has the new type, and move it to the ordinal of the old column
int ordinal = dt.Columns[columnName].Ordinal;
dt.Columns.Add(dc);
dc.SetOrdinal(ordinal);
// Get and convert the values of the old column, and insert them into the new
foreach (DataRow dr in dt.Rows)
{
var obj = dr[columnName];
if (obj != null && obj != DBNull.Value)
{
dr[dc.ColumnName] = Convert.ChangeType(obj, newType);
}
else
{
dr[dc.ColumnName] = obj;
}
}
// Remove the old column
dt.Columns.Remove(columnName);
// Give the new column the old column''s name
dc.ColumnName = columnName;
}
}
public static void ConvertColumnType(this DataSet dataSet, Type sourceType, Type targetType)
{
foreach (DataTable table in dataSet.Tables)
{
var columns = table.Columns.Cast<DataColumn>().Where(x => x.DataType == sourceType).ToList();
//do not use foreach here, otherwise you will encounter "Collection was modified; enumeration operation may not execute."
for (int i = 0; i < columns.Count; i++)
{
table.ConvertColumnType(columns[i].ColumnName, targetType);
}
}
}
关于double的精度的问题
https://www.cnblogs.com/c-primer/p/5992696.html
https://blog.csdn.net/yansmile1/article/details/70145416
涉及到计算机原理中浮点数的存储问题,
float和double的范围是由指数的位数来决定的。
float的指数位有8位,而double的指数位有11位,分布如下:
float:
1bit(符号位)
8bits(指数位)
23bits(尾数位)
double: In c# double is always 8 bytes (64 bits)
1bit(符号位)
11bits(指数位)
52bits(尾数位)
float和double的精度是由尾数的位数来决定的。
浮点数在内存中是按科学计数法来存储的,其整数部分始终是一个隐含着的“1”,由于它是不变的,故不能对精度造成影响。
float:2^23 = 8388608,一共七位,这意味着最多能有7位有效数字,但绝对能保证的为6位,也即float的精度为6~7位有效数字;
double:2^52 = 4503599627370496,一共16位,同理,double的精度为15~16位。
https://stackoverflow.com/questions/1546113/double-to-string-conversion-without-scientific-notation
For a general-purpose¹ solution you need to preserve 339 places:
doubleValue.ToString("0." + new string(''#'', 339))
The maximum number of non-zero decimal digits is 16. 15 are on the right side of the decimal point. The exponent can move those 15 digits a maximum of 324 places to the right. (See the range and precision.)
It works for double.Epsilon
, double.MinValue
, double.MaxValue
, and anything in between.
The performance will be much greater than the regex/string manipulation solutions since all formatting and string work is done in one pass by unmanaged CLR code. Also, the code is much simpler to prove correct.
For ease of use and even better performance, make it a constant:
public static class FormatStrings { public const string DoubleFixedPoint = "0.###################################################################################################################################################################################################################################################################################################################################################"; }
¹ Update: I mistakenly said that this was also a lossless solution. In fact it is not, since ToString
does its normal display rounding for all formats except r
. Live example. Thanks, @Loathing! Please see Lothing’s answer if you need the ability to roundtrip in fixed point notation (i.e, if you’re using .ToString("r")
today).
This solution is not "loseless".
Example:
String t1 = (0.0001/7).ToString("0." + new string(''#'', 339)); // 0.0000142857142857143
versus:
String t2 = (0.0001/7).ToString("r"); // 1.4285714285714287E-05
Precision is lost at the ending decimal places.
用r进行转换
if (value is double d)
{
parameters[2].Value = d.ToString("r");
}
发现是我理解错了,下面2个方法,第一个虽然可以避免科学计数法,但是会丢失精度。
String t1 = (0.0001 / 7).ToString("0." + new string(''#'', 339)); // 0.0000142857142857143 versus:
String t2 = (0.0001 / 7).ToString("r"); // 1.4285714285714287E-05
Console.WriteLine(t1);
Console.WriteLine(t2);
public static class FormatStrings
{
public const string DoubleFixedPoint = "0.###################################################################################################################################################################################################################################################################################################################################################";
public static String ToStandardNotationString(this double d)
{
//Keeps precision of double up to is maximum
return d.ToString("0.#####################################################################################################################################################################################################################################################################################################################################");
}
}
public void DecimalTest()
{
double d = 0.0001 / 7;
Console.WriteLine(d);
Console.WriteLine(d.ToStandardNotationString());
Console.WriteLine(d.ToString("r"));
}
输出
1.42857142857143E-05 科学计数法,小数点后14位。(加上科学计数法的位数,14+5=19位)
0.0000142857142857143 不使用科学计数法,小数点后19位。
1.4285714285714287E-05 //d.ToString("r") 这个小数点16位(16+5=21位置)
c# – 如何使用SqlBulkCopyColumnMappingCollection?
现在我需要做一些映射.我不知道如何创建一个sqlBulkcopyColumnMappingCollection,因为我计划传入映射集合并使用它.但是我不知道该怎么做.我不能做一个新的对象.
这是我现在所在如何添加它做映射放通过?
public void BatchBulkcopy(DataTable dataTable,string DestinationTbl,int batchSize) { // Get the DataTable DataTable dtInsertRows = dataTable; using (sqlBulkcopy sbc = new sqlBulkcopy(connectionString,sqlBulkcopyOptions.KeepIdentity)) { sbc.DestinationTableName = DestinationTbl; // Number of records to be processed in one go sbc.BatchSize = batchSize; // Finally write to server sbc.WritetoServer(dtInsertRows); } }
解决方法
public void BatchBulkcopy(DataTable dataTable,sqlBulkcopyOptions.KeepIdentity)) { sbc.DestinationTableName = DestinationTbl; // Number of records to be processed in one go sbc.BatchSize = batchSize; // Add your column mappings here sbc.columnmappings.Add("field1","field3"); sbc.columnmappings.Add("foo","bar"); // Finally write to server sbc.WritetoServer(dtInsertRows); } }
编辑:
根据意见,目标是制定一般的功能,例如不必在函数中显式地对映射进行硬编码.由于ColumnMappingCollection不能被实例化,我建议传递一个List< string>或类似的包含列映射定义到函数中.例如:
var columnMapping = new List<string>(); columnMapping.Add("field1,field3"); columnMapping.Add("foo,bar");
然后将功能重新定义为
public void BatchBulkcopy(DataTable dataTable,int batchSize,List<string> columnMapping) { // Get the DataTable DataTable dtInsertRows = dataTable; using (sqlBulkcopy sbc = new sqlBulkcopy(connectionString,sqlBulkcopyOptions.KeepIdentity)) { sbc.DestinationTableName = DestinationTbl; // Number of records to be processed in one go sbc.BatchSize = batchSize; // Add your column mappings here foreach(var mapping in columnMapping) { var split = mapping.Split(new[] { ',' }); sbc.columnmappings.Add(split.First(),split.Last()); } // Finally write to server sbc.WritetoServer(dtInsertRows); } }
c# – 让SqlBulkCopy在MiniProfiler中显示为sql
我正在处理的一个问题是由linq生成的重复INSERT语句.
我已将它们转换为sqlBulkcopy命令,但是现在它似乎没有将它记录在MiniProfiler的sql视图中.
是否会为sqlBulkcopy提供相关的命令字符串?
是否可以将批量副本显示在sql命令列表中?
我至少可以在%sql位中计算它吗?
我知道我可以使用MiniProfiler.Current.Step(“正在进行批量复制”),但这不会算作sql,也不会在列表中显示任何细节.
目前的代码如下:
public static void BulkInsertAll<T>(this DataContext dc,IEnumerable<T> entities) { var conn = (dc.Connection as ProfiledDbConnection).InnerConnection as sqlConnection; conn.open(); Type t = typeof(T); var tableAttribute = (TableAttribute)t.GetCustomAttributes( typeof(TableAttribute),false).Single(); var bulkcopy = new sqlBulkcopy(conn) { DestinationTableName = tableAttribute.Name }; //.... bulkcopy.WritetoServer(table); }
解决方法
您可以在the sample project中看到CustomTiming的一些示例用法,其中它用于记录http和redis事件.
如何在sqlBulkcopy中使用它的示例:
string sql = GetBulkcopysql(); // what should show up for the sqlBulkcopy event? using (MiniProfiler.Current.CustomTiming("sqlBulkcopy",sql)) { RunsqlBulkcopy(); // run the actual sqlBulkcopy operation }
Code Sign error: No matching provisioning profile found: Your build settings specify a provisioni...
今天别人让我帮忙调代码,发来工程,运行的时候一直出现如下这个 JB 错误:
点击 “Fix Issue” 还是 TM 不行!!MLGBD,老是提示指定 UUID 的 provisioning profile 找不到,感觉很怪异。我明明重新注册 UDID,重新生成 provisioning profile,并且重新安装,TMD 为毛还不行;百度好多都不给力,只好谷歌;还真通过谷歌找到了答案,虽然是英文的,但是勉强着有道着还是大概知道了原因所在!!参考地址:http://stackoverflow.com/questions/1760518/codesign-error-provisioning-profile-cannot-be-found-after-deleting-expired-prof
大概步骤如下:
1. 找到项目中的 **.xcodeproj 文件,右键打开包内容;
2. 然后找到 project.pbxproj 文件,双击打开;
3. 再然后,搜索 “PROVISIONING_PROFILE", 找到以它开头的都删了
删完以后你就会发现错误提示没了,那么就运行工程,联网情况下 Xcode 会自己
下载并安装 provisioning profile 文件;那么一切都 ok 了!!
最笨的方法是:重新走一步真机测试流程!!嘎嘎,我那 2B 同事就这么干的!!
转载请注明:http://my.oschina.net/LangZiAiFer/admin/new-blog
EntityFramework Core 5.0 VS SQLBulkCopy
【导读】EF Core 5.0伴随着.NET 5.0发布已有一段时日,本节我们来预估当大批量新增数据时,大概是多少区间我们应该考虑SQLBulkCopy而不是EF Core
SQLBulkCopy早出现于.NET Framework 2.0,将数据批量写入利用此类毫无疑问最佳,虽其来源任意,但此类仅适用于SQL Server,每个关系数据库都有其批量处理驱动,这里我们仅仅只讨论SQL Server
性能差异预估批量数据大小
首先给出我们需要用到的测试模型
public class User
{
public int Id { get; set; }
public string Name { get; set; }
public DateTime Birth { get; set; }
public string Email { get; set; }
public string Phone { get; set; }
}
接下来我们则需要模拟数据,为伪造实际生产数据,这里我们介绍一个包Bogus,此包专用来伪造数据,一直在更新从未间断,版本也达到32,如下:
此包中有针对用户类的模拟,具体使用这里就不详细展开,我们构造一个方法来伪造指定数量的用户数据,如下:
private IEnumerable<User> GenerateUseres(int count)
{
var profileGenerator = new Faker<User>()
.RuleFor(p => p.Name, v => v.Person.UserName)
.RuleFor(p => p.Birth, v => v.Person.DateOfBirth)
.RuleFor(p => p.Email, v => v.Person.Email)
.RuleFor(p => p.Phone, v => v.Person.Phone);
return profileGenerator.Generate(count);
}
有了批量伪造数据,接下来我们再利用上下文去新增数据,然后分别打印伪造数据和新增成功所耗费时间,如下:
[HttpPost]
public async Task<IActionResult> GenerateInsert([FromQuery] int count = 1000)
{
var s = new Stopwatch();
s.Start();
var users = GenerateUseres(count);
var gererationTime = s.Elapsed.ToString();
s.Restart();
await _context.Users.AddRangeAsync(users);
var insertedCount = await _context.SaveChangesAsync();
return Ok(new
{
inserted = insertedCount,
generationTime = gererationTime,
insertTime = s.Elapsed.ToString()
});
}
新增100条数据太小,这里我们直接从批量1000条数据开始测试,此时我们将看到所存储数据和实际数据完全一毛一样
通过SQL Server Profiler工具监控得到如下一堆语句
通过运行多次,当然也和笔记本配置有关(i7,6核12线程,内存16G),但还是可以预估批量新增1000条大概耗时为毫秒级,如下:
接下来我们试试新增1万条看看,耗时基本需要1秒,如下:
最后我们再来试试新增十万条数据试试,大概需要14秒才能完成
到了这里想必我们没有必要再往上增长数据,我们来看看利用SqlBulkCopy又将如何
[HttpPost]
public async Task<IActionResult> GenerateAndInsertWithSqlBulkCopy([FromQuery] int count = 1000)
{
var s = new Stopwatch();
s.Start();
var users = GenerateUseres(count);
var gererationTime = s.Elapsed.ToString();
s.Restart();
var dt = new DataTable();
dt.Columns.Add("Id");
dt.Columns.Add("Name");
dt.Columns.Add("Birth");
dt.Columns.Add("Phone");
dt.Columns.Add("Email");
foreach (var user in users)
{
dt.Rows.Add(string.Empty, user.Name, user.Birth, user.Phone, user.Email);
}
using var sqlBulk = new SqlBulkCopy(@"Server=.;Database=EFCore;Trusted_Connection=True;")
{
DestinationTableName = "Users"
};
await sqlBulk.WriteToServerAsync(dt);
return Ok(new
{
inserted = dt.Rows.Count,
generationTime = gererationTime,
insertTime = s.Elapsed.ToString()
});
}
因如上利用EF Core新增时间在毫秒级,那么我们则直接从新增1万条开始测试,如下我们可看到此时与EF Core新增1万条数据差异,耗时远远小于1秒
最后我们再来测试10万条,很显然EF Core耗时结果将为SqlBulkCopy的指数倍(大致14倍,若数据为100万,想想二者其性能差异),如下:
若继续通过SQL Server Profiler监控工具查看SQL语句,很显然SQL语句会很简短,据我所知,SqlBulkCopy是直接将数据通过流形式传输到数据库服务器,然后一次性插入到目标表中,所以性能是杠杠的。
利用SqlBulkCopy和EF Core 5.0,当然理论上不论是EF Core更新到其他任何版本,其性能与SqlBulkCopy不可同日而语,本文我们只是稍加探讨下数据量达到多少时不得不考虑其他手段来处理,而不是利用EF Core新增数据
???? EF Core和SqlBulkCopy性能差异比较,毫无疑问SqlBulkCopy为最佳选手
???? 当新增数据量达到1万+时,若需考虑性能,可采用SqlBulkCopy或其他手段处理数据而不再是EF Core,二者性能差异将呈指数增长
今天关于SqlBulkCopy Precision Error from C# Double to Sql Decimal?的分享就到这里,希望大家有所收获,若想了解更多关于c# – 如何使用SqlBulkCopyColumnMappingCollection?、c# – 让SqlBulkCopy在MiniProfiler中显示为sql、Code Sign error: No matching provisioning profile found: Your build settings specify a provisioni...、EntityFramework Core 5.0 VS SQLBulkCopy等相关知识,可以在本站进行查询。
本文标签: