How to programmatically insert 40000 plus records in MYSQL speedily from Asp.Net

Recently I got a task of importing about 40,000 records for a fixed length delimited file to MY SQL database. Copying the data into a list of Business object was not a problem, but to insert about 40000 records in MYSQL in an asp.net application looked time consuming. More over this event would take place regularly and over 200 times a day and also user need to process the data after it has been uploaded.

My earlier post on multiple insert with Stored procedure seems would not be that effective here as all the data was different in the records.

I first started by using a simple Insert query and then pass the data as parameters. The first efforts completed the job in about 500+ seconds (measurement of the time was being done by the Continued...


Workaround for a major security Issue in Asp.Net till the permanent fix is found
A new vulnerability in asp.net application was found last friday. This vulnerability exists include in all the Asp.net sites, including Asp.Net (All versions), Asp.Net MVC, SharePoint, Any framework in Asp.Net Like Dot Net nuke etc.

An attacker of this vulnerability can use it and download file like web.config which contains sensitive data about the application including connection string for database etc.

Currently a (temporary) workaround (till a proper patch for this vulnerability is made available) has been provided by Microsoft.
The workaround involves setting the customerror in such a way that it returns only one page on any

Continued...

Using Case statement in LINQ to SQL

Working with LINQ I realized that I had to use the simple case statement in my SQL query. There is no special keyword for this. To create a case statement like structure you will have to do it in the select section of the query.

Below is an example of the usage of the case statement in LINQ.

Continued...


All the posts in LINQ series

In Last few weeks I have done a few LINQ series Post. Here is a list of all the posts done.


Filtering data in LINQ with the help of where clause

Using Take and skip keyword to filter records in LINQ


Continued...


Applying aggregate function in LINQ

LINQ also provides with itself important aggregate function. Aggregate function are function that are applied over a sequence like and return only one value like Average, count, sum, Maximum etc…
Below are some of the Aggregate functions provided with LINQ and example of their implementation.

Count

    int[] primeFactorsOf300 = { 2, 2, 3, 5, 5 };

    int uniqueFactors = primeFactorsOf300.Distinct().Count();

The below example provided count for only odd number.

Continued...


LINQ and various types of joins

While working with data most of the time we have to work with relation between different lists of data. Many a times we want to fetch data from both the list at once. This requires us to make different kind of joins between the lists of data.

LINQ support different kinds of join

Inner Join

    List<Customer> customers = GetCustomerList();

    List<Supplier> suppliers = GetSupplierList();

 

Continued...


LINQ and the use of Repeat and Range operator

LINQ is also very useful when it comes to generation of range or repetition of data.  We can generate a range of data with the help of the range method.

    var numbers =

        from n in Enumerable.Range(100, 50)

        select new {Number = n, OddEven = n % 2 == 1 ? "odd" : "even"};


The above query will generate 50 records where the record will start

Continued...

LINQ and Element operators to retrieve specific records

While working with data it’s not always required that we fetch all the records. Many a times we only need to fetch the first record, or some records in some index, in the record set. With LINQ we can get the desired record very easily with the help of the provided element operators.

Simple get the first record.

If you want only the first record in record set we can use the first method [Note that this can also be done easily done with the help of the take method by providing the value as one].

    List<Product> products = GetProductList();

Continued...


Using conversion operators in LINQ to convert result set to desired format

LINQ has a habit of returning things as IEnumerable. But we have all been working with so many other format of lists like array ilist, dictionary etc that most of the time after having the result set we want to get them converted to one of our known format. For this reason LINQ has come up with helper method which can convert the result set in the desired format. Below is an example

var sortedDoubles =

        from d in doubles

        orderby d descending

Continued...


Performing Set operation Distinct, Union, Intersect and Except in LINQ

There are many set operation that are required to be performed while working with any kind of data. This can be done very easily with the help of LINQ methods available for this functionality. Below are some of the examples of the set operation with LINQ.

Finding distinct values in the set of data.

We can use the distinct method to find out distinct values in a given list.

    int[] factorsOf300 = { 2, 2, 3, 5, 5 };

    var uniqueFactors = factorsOf300.Distinct();

Continued...


LINQ And Group keyword for grouping of data

While working with any kind of advanced query grouping is a very important factor. Grouping helps in executing special function like sum, max average etc to be performed on certain groups of data inside the date result set. Grouping is done with the help of the Group method. Below is an example of the basic group functionality.

    int[] numbers = { 5, 4, 1, 3, 9, 8, 6, 7, 2, 0 };

   

    var numberGroups =

Continued...


Using the Order clause to sort the result set

After filtering and retrieving the records most of the time (if not always) we have to sort the record in certain order. The sort order is very important for displaying records or major calculations. In LINQ for sorting data the order keyword is used.

With the help of the order keyword we can decide on the ordering of the result set that is retrieved after the query.  Below is an simple example of the order keyword in LINQ.

    string[] words = { "cherry", "apple", "blueberry" };

    var sortedWords =

Continued...


LINQ Using TakeWhile and SkipWhile to filter records based on condition

In my last post I talked about how to use the take and the Skip keyword to filter out the number of records that we are fetching. But there is only problem with the take and skip statement. The problem lies in the dependency where by the number of records to be fetched has to be passed to it. Many a times the number of records to be fetched is also based on the query itself.

For example if we want to continue fetching records till a certain condition is met on the record set. Let’s say we want to fetch records from the array of number till we get 7. For this kind of query LINQ has exposed the TakeWhile Method.

    int[] numbers = { 5, 4, 1, 3, 9, 8, 6, 7, 2, 0 };

Continued...


Filtering records with the Skip and take keyword in LINQ

In LINQ we can use the take keyword to filter out the number of records that we want to retrieve from the query. Let’s say we want to retrieve only the first 5 records for the list or array then we can use the following query

    int[] numbers = { 5, 4, 1, 3, 9, 8, 6, 7, 2, 0 };

    var first3Numbers = numbers.Take(3);

The TAKE keyword can also be easily applied to list of object in the following way.

var

Continued...

LINQ and where clause to filter out data

LINQ has bought with itself a super power of querying Objects, Database, XML, SharePoint and nearly any other data structure. The power of LINQ lies in the fact that it is managed code that lets you write SQL type code to fetch data. 

Whenever working with data we always need a way to filter out the data based on different condition. In this post we will look at some of the different ways in which we can filter data in LINQ with the help of where clause.

Simple Filter for an array.

Let’s say we have an array of number and we want to filter out data based on some condition. Below is an example

Continued...


Having clause and Grouping Based on condition in LINQ

Hi,

While querying with LINQ, some times we will have to use the group By clause. But many a times we also want to use the having clause of SQL with the group by clause in the LINQ. There is no direct having keyword in LINQ to do this we need to use the where clause itself.

You can query the LINQ to SQL to have both Group By and Having like this.var categories =   from p in db.Products
               group p by p.CategoryID into g
               where g.Count() >= 10
               select new {
                      g.Key,
             

Continued...

LINQ queries and IN clause in SQL

Hi,

When working with the LINQ queries for SQL one of the common queries that we need to run is the select query with IN clause. In SQL IN clause is used to provide more than one value to match in the where clause.

Something like the query below

Select * from Table
where column1 in (‘Value1’, ‘Value2’, ‘Value3’)

To do a similar query in LINQ we can use

var list = from t in table
                where t.column1 = Value1’
                And t.column1 = Value2’
                t.column1 = Value3’
                Select t

 But what if the

Continued...

Improvements made in Base class libraries in Dot net framework 4.0

Hi,