Prediction task on Order Volume

Before we get into analyze part, we need to do data preprocessing part first.

Data Observe

1. We are dealing with order volume of 7 different warehouses. It is obvious that those order volumes are related to time. Companies have different volumes in different time. It may be good for us to treat those order volume as time series data.
2. We need to first have a board look at data including the date range in each warehouse. To avoid the case that the time of data in each warehouse is not consistent, which will lead to different amount of data size.

Data preprocessing

1. Missing Date

   1. Raw data always contains missing value in real life. We don’t know that at very first time. It is necessary for us to test whether there is any missing value stay in each warehouses.

      1. Writing function check_data_missing , we could easily check whether there are missing value in different warehouses. To achieve that, we need to analyze one warehouse at a time.

      2. One technique that can be used in checking missing data in time series data is date_range. It allows us to generate all date within the range we set, and it can generate the date in the frequency we want. Frequency includes day, month and year. By using this, we can compare date in raw data to the date generated by date_range.

      3. def check_data_missing(train_data_raw):
             for warehouse in train_data_raw['warehouse'].unique():
                 temp_pd = train_data_raw[train_data_raw['warehouse'] == warehouse]
                 missing_date_time = pd.date_range(
                     start=temp_pd['date'].min(), end=temp_pd['date'].max()
                 ).difference(temp_pd.date)
                 print(f'missing time:{Style.BRIGHT}{Fore.YELLOW}{missing_date_time}{Style.RESET_ALL}')
         

2. Fill out Missing Value

   1. Still, we fill out each warehouse at a time. In this project, I used mean value of previous three days to fill out the missing value.

   2. We set the window size equal 3

   3. def padding_data(df, miss_date, warehouse):
          temp_df = df[df['warehouse'] == warehouse]
          temp_df.set_index('date', inplace=True)
          window_size = 3
                
          window_start = miss_date - pd.Timedelta(days=window_size)
          window_end = miss_date + pd.Timedelta(days=1)
          window_data = temp_df[(temp_df.index >= window_start) & (temp_df.index < window_end)]
                
          estimated_order = window_data['orders'].mean() if len(window_data) > 0 else None
          print(f"The estimated order quantity for {miss_date} is {estimated_order}")
                
          return estimated_order
      

   4. pd.Timedelta allows us to calculate the time gaps. pd.Timedelta(days=3) would return “3 days 00:00:00”. By using this, we can find out the range of date and calculate its mean .

   5. Data Observe & Fill out Missing Value

      1. for warehouse in train_data_raw['warehouse'].unique():
             print(f'{Style.BRIGHT}{Fore.BLUE}**{warehouse}**{Style.RESET_ALL}')
             temp_pd = train_data_raw[train_data_raw['warehouse'] == warehouse]
             train_time_period = temp_pd['date'].max() - temp_pd['date'].min()
             print(f'{warehouse} firstday :{temp_pd.date.min()}')
             print(f'{warehouse}last day : {temp_pd.date.max()}')
             print(f'{warehouse}Date period:{train_time_period}')
                      
             missing_date_time = pd.date_range(
                 start= temp_pd['date'].min(), end=temp_pd['date'].max()).difference(temp_pd.date)
             print(f'missing time:{Style.BRIGHT}{Fore.YELLOW}{missing_date_time}{Style.RESET_ALL}')
                  
             for miss_date in list(missing_date_time):
                 print(f"Dealing with{warehouse}'s'{miss_date}日期")
                 add_df = pd.DataFrame(
                     {
                         'warehouse': [warehouse],
                         'date': [miss_date],
                         'orders': [padding_data(train_data_raw, miss_date, warehouse)],
                         'holiday_name': [0],
                         'holiday': [0],
                         "shutdown": [0],
                         "mini_shutdown": [0],
                         "shops_closed": [0],
                         "winter_school_holidays": [0],
                         "school_holidays": [0],
                         "blackout": [0],
                         "mov_change": [0],
                         "frankfurt_shutdown": [0],
                         "precipitation": [0],
                         "snow": [0],
                         "user_activity_1": [0],
                         "user_activity_2": [0],
                         "id": [0],
                         "holiday_mark": [0],
                     })
                 compare_df = train_data_raw
                 #train_data_raw = pd.concat([train_data_raw, add_df], ignore_index=True)
                 train_data_raw = train_data_raw._append(add_df, ignore_index=True)
                 train_data_raw.groupby('warehouse')
                  
                  
                 contains_date = miss_date in set(train_data_raw['date'])
         

      2. Some columns are not numerical variable, then we need to use another way to fill them out. The most simple way is to fill out “None”

         1. train_data_raw['holiday_name'] = train_data_raw['holiday_name'].fillna('None')
            

   6. Date time processing

      1. Because it is a time series data, it is really important to figure out what impact date will affect on the data. We need to be more specific by splitting the time out.

      2. train_data_raw['year'] = train_data_raw['date'].dt.year
         train_data_raw['month'] = train_data_raw['date'].dt.month
         train_data_raw['week'] = train_data_raw['date'].dt.isocalendar().week
         train_data_raw['day'] = train_data_raw['date'].dt.day