论文复现： DSIN-阿里点击率预测三部曲-3

本文介绍了基于PaddlePaddle复现Deep Session Interest Network（DSIN）用于点击率预测的项目。项目使用阿里Ali_Display_Ad_Click数据集，将用户历史交互数据划分为会话建模兴趣。复现的DSIN模型测试AUC达0.6356，详述了数据集预处理、模型结构、代码构成、训练测试流程及复现中遇到的问题与解决心得。

一、简介

Deep Session Interest Network for Click-Through Rate Prediction 是点击率预测问题的一篇经典论文，该论文的先前工作有大家非常熟悉的DIN，DIEN。都是关注于用户的兴趣，对用户的历史会话行为建模出用户的兴趣表示。DSIN模型观察到了，用户在 session 中的兴趣是高度相近的，但是在不同的 session 中的兴趣是不同的，如下图所示： 

根据上述观察，DSIN将用户的历史交互数据划分成了一个个 session ， 然后再通过自注意力和双向LSTM对用户的 session 兴趣进行建模。模型框架与DIN，DIEN类似，如下图所示： 论文连接：Deep Session Interest Network for Click-Through Rate Prediction

二、复现精度

基于paddlepaddle深度学习框架，对文献算法进行复现后，本项目达到的测试精度，如下表所示。

参数设置可详见config_bigdata.yaml文件。

三、数据集

本项目所使用的数据集Ali_Display_Ad_Click是由阿里所提供的一个淘宝展示广告点击率预估数据集。

1、原始数据集介绍

原始样本骨架raw_sample：淘宝 中随机抽样了114万用户8天内的广告展示/点击日志（2600万条记录），构成原始的样本骨架user：脱敏过的用户ID；adgroup_id：脱敏过的广告单元ID；time_stamp：时间戳；pid：资源位；nonclk：为1代表没有点击；为0代表点击；clk：为0代表没有点击；为1代表点击；

user,time_stamp,adgroup_id,pid,nonclk,clk581738,1494137644,1,430548_1007,1,0

登录后复制广告基本信息表ad_feature：本数据集涵盖了raw_sample中全部广告的基本信息adgroup_id：脱敏过的广告ID；cate_id：脱敏过的商品类目ID；campaign_id：脱敏过的广告计划ID；customer: 脱敏过的广告主ID；brand：脱敏过的品牌ID；price: 宝贝的价格

adgroup_id,cate_id,campaign_id,customer,brand,price63133,6406,83237,1,95471,170.0

登录后复制用户基本信息表user_profile：本数据集涵盖了raw_sample中全部用户的基本信息userid：脱敏过的用户ID；cms_segid：微群ID；cms_group_id：cms_group_id；final_gender_code：性别 1:男,2:女；age_level：年龄层次； 1234pvalue_level：消费档次，1:低档，2:中档，3:高档；shopping_level：购物深度，1:浅层用户,2:中度用户,3:深度用户occupation：是否大学生 ，1:是,0:否new_user_class_level：城市层级

userid,cms_segid,cms_group_id,final_gender_code,age_level,pvalue_level,shopping_level,occupation,new_user_class_level 234,0,5,2,5,,3,0,3

登录后复制用户的行为日志behavior_log：本数据集涵盖了raw_sample中全部用户22天内的购物行为user：脱敏过的用户ID；time_stamp：时间戳；btag：行为类型, 包括以下四种：(pv:浏览),(cart:加入购物车),(fav:喜欢),(buy:购买)cate：脱敏过的商品类目id；brand: 脱敏过的品牌id；

user,time_stamp,btag,cate,brand558157,1493741625,pv,6250,91286

登录后复制

预处理数据集介绍

对原始数据集中的四个文件，参考原论文的数据预处理过程对数据进行处理，形成满足DSIN论文条件且可以被reader直接读取的数据集。 数据集共有八个pkl文件，训练集和测试集各自拥有四个，以训练集为例，这四个文件为train_feat_input.pkl、train_sess_input、train_sess_length和train_label.pkl。各自存储了按0.25的采样比进行采样后的user及item特征输入，用户会话特征输入、用户会话长度和标签数据。

四、环境依赖

硬件：

x86 cpuNVIDIA GPU

框架：

PaddlePaddle = 2.2.2Python = 3.7

其他依赖项：

PaddleRec

五、快速开始

1、克隆PaddleRec

In [1]

#clone PaddleRecimport os!ls /home/aistudio/data/!ls work/!python --version!pip list | grep paddlepaddleif not os.path.isdir('work/PaddleRec'):    !cd work && git clone https://gitee.com/paddlepaddle/PaddleRec.git

登录后复制

data131207PaddleRecPython 3.7.4paddlepaddle-gpu       2.2.2.post101

登录后复制

2、解压数据集并移动到PaddleRec的datasets里

In [2]

#解压数据集!tar -zxvf data/data131207/model_input.tar.gz!mkdir '/home/aistudio/work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/'!mkdir '/home/aistudio/work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/'!mkdir '/home/aistudio/work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/'!mv model_input/test_feat_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/!mv model_input/test_label.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/!mv model_input/test_sess_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/!mv model_input/test_session_length.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/!mv model_input/train_feat_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/!mv model_input/train_label.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/!mv model_input/train_sess_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/!mv model_input/train_session_length.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/

登录后复制

model_input/model_input/test_session_length.pklmodel_input/test_sess_input.pklmodel_input/train_sess_input.pklmodel_input/train_feat_input.pklmodel_input/test_feat_input.pklmodel_input/test_label.pklmodel_input/train_label.pklmodel_input/train_session_length.pkl

登录后复制

3、写入模型相关代码

In [3]

!mkdir '/home/aistudio/work/PaddleRec/models/rank/dsin'%cd '/home/aistudio/work/PaddleRec/models/rank/dsin'

登录后复制

/home/aistudio/work/PaddleRec/models/rank/dsin

登录后复制In [4]

%%writefile net.py# Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at##     https://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.import paddleimport paddle.nn as nnimport paddle.nn.functional as Fimport mathimport numpy as npfrom sequence_layers import PositionalEncoder, AttentionSequencePoolingLayer, MLPclass DSIN_layer(nn.Layer):    def __init__(self, user_size, adgroup_size, pid_size, cms_segid_size, cms_group_size,                 final_gender_size, age_level_size, pvalue_level_size, shopping_level_size,                 occupation_size, new_user_class_level_size, campaign_size,customer_size, cate_size, brand_size,  # above is all sparse feat size                 sparse_embed_size = 4, att_embedding_size = 8, sess_count = 5, sess_max_length = 10, l2_reg_embedding=1e-6):        super().__init__()        # feature size        self.user_size = user_size        self.adgroup_size = adgroup_size           self.pid_size = pid_size        self.cms_segid_size = cms_segid_size        self.cms_group_size = cms_group_size        self.final_gender_size = final_gender_size        self.age_level_size = age_level_size        self.pvalue_level_size = pvalue_level_size        self.shopping_level_size = shopping_level_size        self.occupation_size = occupation_size        self.new_user_class_level_size = new_user_class_level_size        self.campaign_size = campaign_size        self.customer_size = customer_size        self.cate_size = cate_size        self.brand_size = brand_size        # sparse embed size        self.sparse_embed_size = sparse_embed_size        # transform attention embed size        self.att_embedding_size = att_embedding_size        # hyper_parameters        self.sess_count = 5        self.sess_max_length = 10        # sparse embedding layer        self.userid_embeddings_var = paddle.nn.Embedding(            self.user_size,            self.sparse_embed_size,            sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.adgroup_embeddings_var = paddle.nn.Embedding(            self.adgroup_size,            self.sparse_embed_size,            sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.pid_embeddings_var = paddle.nn.Embedding(            self.pid_size,            self.sparse_embed_size,            #sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.cmsid_embeddings_var = paddle.nn.Embedding(            self.cms_segid_size,            self.sparse_embed_size,            #sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.cmsgroup_embeddings_var = paddle.nn.Embedding(            self.cms_group_size,            self.sparse_embed_size,            #sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.gender_embeddings_var = paddle.nn.Embedding(            self.final_gender_size,            self.sparse_embed_size,            #sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.age_embeddings_var = paddle.nn.Embedding(            self.age_level_size,            self.sparse_embed_size,            #sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.pvalue_embeddings_var = paddle.nn.Embedding(            self.pvalue_level_size,            self.sparse_embed_size,            #sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.shopping_embeddings_var = paddle.nn.Embedding(            self.shopping_level_size,            self.sparse_embed_size,            #sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.occupation_embeddings_var = paddle.nn.Embedding(            self.occupation_size,            self.sparse_embed_size,            #sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.new_user_class_level_embeddings_var = paddle.nn.Embedding(            self.new_user_class_level_size,            self.sparse_embed_size,            #sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.campaign_embeddings_var = paddle.nn.Embedding(            self.campaign_size,            self.sparse_embed_size,            sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.customer_embeddings_var = paddle.nn.Embedding(            self.customer_size,            self.sparse_embed_size,            sparse=True,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.cate_embeddings_var = paddle.nn.Embedding(            self.cate_size,            self.sparse_embed_size,            sparse=True,            padding_idx=0,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        self.brand_embeddings_var = paddle.nn.Embedding(            self.brand_size,            self.sparse_embed_size,            sparse=True,            padding_idx=0,            weight_attr=paddle.ParamAttr(                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))        # sess interest extractor layer        self.position_encoding = PositionalEncoder(2*self.sparse_embed_size)        self.transform = nn.TransformerEncoderLayer(            d_model = self.att_embedding_size,             nhead = 8,            dim_feedforward = 64,            weight_attr = self._get_weight_attr(),            bias_attr= False,            dropout = 0.0)        # sess interest interacting layer        self.bilstm = nn.LSTM(2*self.sparse_embed_size, 2*self.sparse_embed_size, num_layers = 2, direction='bidirectional')        # sess interest activating layer        self.transform_actpool = AttentionSequencePoolingLayer(weight_normalization=True, name='transform')        self.lstm_actpool = AttentionSequencePoolingLayer(weight_normalization=True, name='lstm')        # MLP moudle        self.mlp = MLP(mlp_hidden_units=[77, 200, 80])    def _get_weight_attr(self):        return paddle.ParamAttr(initializer=nn.initializer.TruncatedNormal(std=0.05))    def forward(self, inputs):        '''        inputs : tulpe, (sparse_input, dense_input, sess_input, sess_length)            sparse_input: (N, 15)            dense_input: (N,)            sess_input：(N, 10, 10)            sess_length: (N,)        '''        sparse_input, dense_input, sess_input, sess_length = inputs        #assert(type(sess_length) == paddle.Tensor), f"At Attention SequencePoolingLayer expected inputs[2]'s type is paddle.Tensor, but got {type(sess_length)}"        # sparse and dense feature        self.user = sparse_input[:, 0]        self.adgroup = sparse_input[:, 1]        self.pid = sparse_input[:, 2]        self.cmsid = sparse_input[:, 3]        self.cmsgroup = sparse_input[:, 4]        self.gender = sparse_input[:, 5]        self.age = sparse_input[:, 6]        self.pvalue = sparse_input[:, 7]        self.shopping = sparse_input[:, 8]        self.occupation = sparse_input[:, 9]        self.new_user_class = sparse_input[:, 10]        self.campaign = sparse_input[:, 11]        self.customer = sparse_input[:, 12]        self.cate = sparse_input[:, 13]        self.brand = sparse_input[:, 14]        self.price = dense_input.unsqueeze_(-1)        # sparse feature embedding        self.user_embeded = self.userid_embeddings_var(self.user)        self.adgroup_embeded = self.adgroup_embeddings_var(self.adgroup)        self.pid_embeded = self.pid_embeddings_var(self.pid)        self.cmsid_embeded = self.cmsid_embeddings_var(self.cmsid)        self.cmsgroup_embeded = self.cmsgroup_embeddings_var(self.cmsgroup)        self.gender_embeded = self.gender_embeddings_var(self.gender)        self.age_embeded = self.age_embeddings_var(self.age)        self.pvalue_embeded = self.pvalue_embeddings_var(self.pvalue)        self.shopping_embeded = self.shopping_embeddings_var(self.shopping)        self.occupation_embeded = self.occupation_embeddings_var(self.occupation)        self.new_user_class_embeded = self.new_user_class_level_embeddings_var(self.new_user_class)        self.campaign_embeded = self.campaign_embeddings_var(self.campaign)        self.customer_embeded = self.customer_embeddings_var(self.customer)        self.cate_embeded = self.cate_embeddings_var(self.cate)        self.brand_embeded = self.brand_embeddings_var(self.brand)        # concat query embeded          # Note: query feature is cate_embeded and brand_embeded        query_embeded = paddle.concat([self.cate_embeded,self.brand_embeded],-1)        # concat sparse feature embeded          deep_input_embeded = paddle.concat([self.user_embeded, self.adgroup_embeded, self.pid_embeded, self.cmsid_embeded,                                    self.cmsgroup_embeded, self.gender_embeded, self.age_embeded, self.pvalue_embeded,                                    self.shopping_embeded, self.occupation_embeded, self.new_user_class_embeded,                                    self.campaign_embeded, self.customer_embeded, self.cate_embeded, self.brand_embeded], -1)        # sess_interest_division part        #cate_sess_embeded = self.cate_embeddings_var(paddle.to_tensor(sess_input[:, ::2, :]))        #brand_sess_embeded = self.brand_embeddings_var(paddle.to_tensor(sess_input[:, 1::2, :]))        cate_sess_embeded = self.cate_embeddings_var(sess_input[:, ::2, :])        brand_sess_embeded = self.brand_embeddings_var(sess_input[:, 1::2, :])        # tr_input (n,5,10,8)        tr_input = paddle.concat([cate_sess_embeded,brand_sess_embeded],axis=-1)         # sess interest extractor part        lstm_input = []        for i in range(self.sess_count):            tr_sess_input = self.position_encoding( tr_input[:, i, :, :] )            tr_sess_input = self.transform(tr_sess_input)            tr_sess_input = paddle.mean(tr_sess_input, axis=1, keepdim=True)            lstm_input.append(tr_sess_input)        lstm_input = paddle.concat([lstm_input[0], lstm_input[1], lstm_input[2], lstm_input[3], lstm_input[4]], axis=1)        lstm_output, _ = self.bilstm(lstm_input)        lstm_output = (lstm_output[:, :, :2*self.sparse_embed_size] + lstm_output[:, :, 2*self.sparse_embed_size:])/2        # sess interest activating layer        lstm_input = self.transform_actpool([query_embeded, lstm_input, sess_length])        lstm_output = self.lstm_actpool([query_embeded, lstm_output, sess_length])        # concatenate all moudle output        mlp_input = paddle.concat([deep_input_embeded, paddle.nn.Flatten()(lstm_input), paddle.nn.Flatten()(lstm_output), self.price], axis=-1)        out = self.mlp(mlp_input)        return out

登录后复制

Writing net.py

登录后复制In [5]

%%writefile sequence_layers.py# Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at##     https://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.import paddleimport paddle.nn as nnimport numpy as npimport copyimport mathclass PositionalEncoder(nn.Layer):    def __init__(self, d_model, max_seq_len=50):        #d_model为嵌入维度        super(PositionalEncoder, self).__init__()        self.d_model = d_model        position = np.array([[pos / np.power(10000, 2. * i / self.d_model)                            for i in range(self.d_model)]                            for pos in range(max_seq_len)])         # Second part, apply the cosine to even columns and sin to odds.        position[:, 0::2] = np.sin(position[:, 0::2])  # dim 2i        position[:, 1::2] = np.cos(position[:, 1::2])  # dim 2i+1        self.position = self.create_parameter(shape=[max_seq_len,self.d_model],                                            default_initializer=paddle.nn.initializer.Assign(value=position))    def forward(self, x):        x = x*math.sqrt(self.d_model)        seq_len = x.shape[1]        x = x+self.position[:seq_len,:]        return xclass AttentionSequencePoolingLayer(nn.Layer):    def __init__(self, dnn_units=[8, 64, 16], dnn_activation='sigmoid', weight_normalization=False, name=None):        super().__init__()        self.dnn_units = dnn_units        self.dnn_activation = 'sigmoid'        self.weight_normalization = weight_normalization        self.name = name        layer_list = []        #bn_list = []        for i in range(len(dnn_units)-1):            dnn_layer = nn.Linear(                in_features = self.dnn_units[i] if i != 0 else self.dnn_units[i]*4 ,                out_features = self.dnn_units[i+1],                  weight_attr= self._weight_init())            self.add_sublayer(self.name + f'linear_{i}', dnn_layer)            layer_list.append(dnn_layer)            #layer_list.append(copy.deepcopy(dnn_layer))            #bn_layer = nn.BatchNorm(50)            #self.add_sublayer(self.name + f'bn_{i}', bn_layer)            #bn_list.append(bn_layer)            #bn_list.append(copy.deepcopy(bn_layer))        #self.bn_layer = nn.LayerList(bn_list)        self.layers = nn.LayerList(layer_list)        self.dnn = nn.Linear(self.dnn_units[-1], 1, weight_attr=self._weight_init())         self.activation = nn.Sigmoid()        self.soft = nn.Softmax()    def _weight_init(self):        return paddle.framework.ParamAttr(initializer=paddle.nn.initializer.XavierNormal())    def forward(self, inputs):        querys, keys, sess_length = inputs        #assert(type(sess_length) == paddle.Tensor), f"At Attention SequencePoolingLayer expected inputs[2]'s type is paddle.Tensor, but got {type(sess_length)}"        keys_length = keys.shape[1]        key_masks = nn.functional.sequence_mask(sess_length, keys_length)         querys = paddle.tile(querys.unsqueeze(1), [1, keys_length, 1])        att_input = paddle.concat([querys, keys, querys-keys, querys*keys], axis=-1)        for i, layer in enumerate(self.layers):            att_input = layer(att_input)            #att_input = self.bn_layer[i](att_input)  # BatchNomalization            att_input = self.activation(att_input) # activation         att_score = self.dnn(att_input)  # (N, 50, 1)        att_score = paddle.transpose(att_score, [0, 2, 1]) # (N, 1, 50)        if self.weight_normalization:            paddings = paddle.ones_like(att_score) * (-2 ** 32 + 1)        else:            paddings = paddle.zeros_like(att_score)        att_score = paddle.where(key_masks.unsqueeze(1) == 1, att_score, paddings)  # key_masks.unsqueeze in order to keep shape same as att_score        att_score = self.soft(att_score)        out = paddle.matmul(att_score, keys)        return outclass MLP(nn.Layer):    def __init__(self, mlp_hidden_units, use_bn=True):        super().__init__()        self.mlp_hidden_units = mlp_hidden_units        self.acitivation = paddle.nn.Sigmoid()        layer_list = []        for i in range(len(mlp_hidden_units)-1):            dnn_layer = nn.Linear(                in_features = self.mlp_hidden_units[i],                out_features = self.mlp_hidden_units[i+1],                  weight_attr= self._weight_init())            self.add_sublayer(f'linear_{i}', dnn_layer)            layer_list.append(dnn_layer)        self.layers = nn.LayerList(layer_list)        self.dense = nn.Linear(self.mlp_hidden_units[-1], 1, bias_attr=True, weight_attr= self._weight_init())        self.predict_layer = nn.Sigmoid()    def _weight_init(self):        return paddle.framework.ParamAttr(initializer=paddle.nn.initializer.XavierNormal())    def forward(self, x):        for layer in self.layers:            x = layer(x)            x = self.acitivation(x)        x = self.dense(x)        x = self.predict_layer(x)        return x

登录后复制

Writing sequence_layers.py

登录后复制In [6]

%%writefile dygraph_model.py# Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at##     https://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.import paddleimport paddle.nn as nnimport paddle.nn.functional as Fimport mathimport netclass DygraphModel():    # define model    def create_model(self, config):        user_size = config.get("hyper_parameters.user_size")        cms_segid_size = config.get("hyper_parameters.cms_segid_size")        cms_group_size = config.get("hyper_parameters.cms_group_size")        final_gender_size = config.get(            "hyper_parameters.final_gender_size")        age_level_size = config.get("hyper_parameters.age_level_size")        pvalue_level_size = config.get("hyper_parameters.pvalue_level_size")        shopping_level_size = config.get(            "hyper_parameters.shopping_level_size")        occupation_size = config.get("hyper_parameters.occupation_size")        new_user_class_level_size = config.get(            "hyper_parameters.new_user_class_level_size")        adgroup_size = config.get("hyper_parameters.adgroup_size")        cate_size = config.get("hyper_parameters.cate_size")        campaign_size = config.get("hyper_parameters.campaign_size")        customer_size = config.get("hyper_parameters.customer_size")        brand_size = config.get("hyper_parameters.brand_size")        pid_size = config.get("hyper_parameters.pid_size")        feat_embed_size = config.get(            "hyper_parameters.feat_embed_size")        dsin_model = net.DSIN_layer(            user_size, adgroup_size, pid_size, cms_segid_size, cms_group_size,            final_gender_size, age_level_size, pvalue_level_size, shopping_level_size,            occupation_size, new_user_class_level_size, campaign_size, customer_size,            cate_size, brand_size, sparse_embed_size=feat_embed_size, l2_reg_embedding=1e-6)        return dsin_model    # define loss function by predicts and label    def create_loss(self, pred, label):        return paddle.nn.BCELoss()(pred,label)    # define feeds which convert numpy of batch data to paddle.tensor    def create_feeds(self, batch_data, config):        data, label = (batch_data[0], batch_data[1], batch_data[2], batch_data[3]), batch_data[-1]        #data, label = batch_data[0], batch_data[1]        label = label.reshape([-1,1])        return label, data    # define optimizer    def create_optimizer(self, dy_model, config):        lr = config.get("hyper_parameters.optimizer.learning_rate", 0.001)        optimizer = paddle.optimizer.Adam(            learning_rate=lr, parameters=dy_model.parameters())        return optimizer    # define metrics such as auc/acc    # multi-task need to define multi metric    def create_metrics(self):        metrics_list_name = ["auc"]        auc_metric = paddle.metric.Auc("ROC")        metrics_list = [auc_metric]        return metrics_list, metrics_list_name    # construct train forward phase    def train_forward(self, dy_model, metrics_list, batch_data, config):        label, input_tensor = self.create_feeds(batch_data, config)        pred = dy_model.forward(input_tensor)        # update metrics        predict_2d = paddle.concat(x=[1 - pred, pred], axis=1)        metrics_list[0].update(preds=predict_2d.numpy(), labels=label.numpy())        loss = self.create_loss(pred,paddle.cast(label, "float32"))        print_dict = {'loss': loss}        # print_dict = None        return loss, metrics_list, print_dict    def infer_forward(self, dy_model, metrics_list, batch_data, config):        label, input_tensor = self.create_feeds(batch_data, config)        pred = dy_model.forward(input_tensor)        # update metrics        predict_2d = paddle.concat(x=[1 - pred, pred], axis=1)        metrics_list[0].update(preds=predict_2d.numpy(), labels=label.numpy())        return metrics_list, None

登录后复制

Writing dygraph_model.py

登录后复制In [7]

%%writefile dsin_reader.py# Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at##     https://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.from __future__ import print_functionimport numpy as npfrom paddle.io import IterableDatasetimport pandas as pdsparse_features = ['userid', 'adgroup_id', 'pid', 'cms_segid', 'cms_group_id', 'final_gender_code', 'age_level',                    'pvalue_level', 'shopping_level', 'occupation', 'new_user_class_level ', 'campaign_id',                    'customer', 'cate_id', 'brand']dense_features = ['price']class RecDataset(IterableDataset):    def __init__(self, file_list, config):        super().__init__()        self.file_list = file_list        data_file = [ f.split('/')[-1] for f in file_list]        mode = data_file[0].split('_')[0]        data_dir = file_list[0].split(data_file[0])[0]        assert(mode == 'train' or mode == 'test' or mode == 'sample'), f"mode must be 'train' or 'test', but get '{mode}'"        feat_input = pd.read_pickle(data_dir + mode + '_feat_input.pkl')        self.sess_input = pd.read_pickle(data_dir + mode + '_sess_input.pkl')        self.sess_length = pd.read_pickle(data_dir + mode + '_session_length.pkl')        self.label = pd.read_pickle(data_dir + mode + '_label.pkl')        if str(type(self.label)).split("'")[1] != 'numpy.ndarray':            self.label = self.label.to_numpy()        self.label = self.label.astype('int64')        self.num_samples = self.label.shape[0]        self.sparse_input = feat_input[sparse_features].to_numpy().astype('int64')        self.dense_input = feat_input[dense_features].to_numpy().reshape(-1).astype('float32')    def __iter__(self):        for i in range(self.num_samples):            yield [self.sparse_input[i, :], self.dense_input[i], self.sess_input[i, :, :], self.sess_length[i], self.label[i]]

登录后复制

Writing dsin_reader.py

登录后复制In [8]

%%writefile config_bigdata.yaml# Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved.## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at##     https://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.runner:  train_data_dir: "../../../datasets/Ali_Display_Ad_Click_DSIN/big_train/"  train_reader_path: "dsin_reader" # importlib format  use_gpu: True  use_auc: True  train_batch_size: 4096  epochs: 1  print_interval: 50  model_save_path: "output_model_all_dsin"  test_data_dir: "../../../datasets/Ali_Display_Ad_Click_DSIN/big_test/"  infer_reader_path: "dsin_reader" # importlib format  infer_batch_size: 16384 # 2**14  infer_load_path: "output_model_all_dsin"  infer_start_epoch: 0  infer_end_epoch: 1# hyper parameters of user-defined networkhyper_parameters:  # optimizer config  optimizer:    class: Adam    learning_rate: 0.00235  # user feature size  user_size: 265442  cms_segid_size: 97  cms_group_size: 13  final_gender_size: 2  age_level_size: 7  pvalue_level_size: 4  shopping_level_size: 3  occupation_size: 2  new_user_class_level_size: 5  # item feature size  adgroup_size: 512431  cate_size: 11859   #max value + 1  campaign_size: 309448  customer_size: 195841  brand_size: 362855  #max value + 1  # context feature size  pid_size: 2  # embedding size  feat_embed_size: 4

登录后复制

Writing config_bigdata.yaml

登录后复制

4、利用PaddleRec的trainer以及infer进行模型训练及其测试

In [9]

!python ../../../tools/trainer.py -m config_bigdata.yaml

登录后复制

2024-05-11 19:50:56,823 - INFO - **************common.configs**********2024-05-11 19:50:56,823 - INFO - use_gpu: True, use_xpu: False, use_visual: False, train_batch_size: 4096, train_data_dir: ../../../datasets/Ali_Display_Ad_Click_DSIN/big_train/, epochs: 1, print_interval: 50, model_save_path: output_model_all_dsin2024-05-11 19:50:56,823 - INFO - **************common.configs**********W0511 19:50:56.825248  1525 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1W0511 19:50:56.831076  1525 device_context.cc:465] device: 0, cuDNN Version: 7.6.2024-05-11 19:51:01,867 - INFO - read data2024-05-11 19:51:01,867 - INFO - reader path:dsin_reader2024-05-11 19:51:13,903 - INFO - epoch: 0, batch_id: 0, auc:0.502794, loss:0.85580873, avg_reader_cost: 0.00291 sec, avg_batch_cost: 0.01317 sec, avg_samples: 81.92000, ips: 6220.65504 ins/s2024-05-11 19:51:33,319 - INFO - epoch: 0, batch_id: 50, auc:0.495701, loss:0.19559237, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38773 sec, avg_samples: 4096.00000, ips: 10564.02249 ins/s2024-05-11 19:51:52,451 - INFO - epoch: 0, batch_id: 100, auc:0.499694, loss:0.21434923, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38206 sec, avg_samples: 4096.00000, ips: 10720.87298 ins/s2024-05-11 19:52:10,842 - INFO - epoch: 0, batch_id: 150, auc:0.512509, loss:0.19038938, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.36725 sec, avg_samples: 4096.00000, ips: 11153.31692 ins/s2024-05-11 19:52:28,755 - INFO - epoch: 0, batch_id: 200, auc:0.530944, loss:0.20696387, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.35769 sec, avg_samples: 4096.00000, ips: 11451.33054 ins/s2024-05-11 19:52:46,030 - INFO - epoch: 0, batch_id: 250, auc:0.545280, loss:0.18852976, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.34493 sec, avg_samples: 4096.00000, ips: 11874.79419 ins/s2024-05-11 19:53:03,111 - INFO - epoch: 0, batch_id: 300, auc:0.558348, loss:0.20377612, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34106 sec, avg_samples: 4096.00000, ips: 12009.68762 ins/s2024-05-11 19:53:20,102 - INFO - epoch: 0, batch_id: 350, auc:0.567205, loss:0.2231454, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33924 sec, avg_samples: 4096.00000, ips: 12073.90980 ins/s2024-05-11 19:53:36,952 - INFO - epoch: 0, batch_id: 400, auc:0.572662, loss:0.2543741, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33644 sec, avg_samples: 4096.00000, ips: 12174.55680 ins/s2024-05-11 19:53:54,328 - INFO - epoch: 0, batch_id: 450, auc:0.577503, loss:0.16823483, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34696 sec, avg_samples: 4096.00000, ips: 11805.51984 ins/s2024-05-11 19:54:13,481 - INFO - epoch: 0, batch_id: 500, auc:0.580811, loss:0.19309358, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38248 sec, avg_samples: 4096.00000, ips: 10709.07133 ins/s2024-05-11 19:54:32,650 - INFO - epoch: 0, batch_id: 550, auc:0.584353, loss:0.19425544, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38280 sec, avg_samples: 4096.00000, ips: 10700.23452 ins/s2024-05-11 19:54:51,018 - INFO - epoch: 0, batch_id: 600, auc:0.587535, loss:0.19358435, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.36678 sec, avg_samples: 4096.00000, ips: 11167.49886 ins/s2024-05-11 19:55:08,682 - INFO - epoch: 0, batch_id: 650, auc:0.590837, loss:0.21790585, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.35272 sec, avg_samples: 4096.00000, ips: 11612.52946 ins/s2024-05-11 19:55:26,055 - INFO - epoch: 0, batch_id: 700, auc:0.594234, loss:0.19218928, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34689 sec, avg_samples: 4096.00000, ips: 11807.69064 ins/s2024-05-11 19:55:43,041 - INFO - epoch: 0, batch_id: 750, auc:0.597527, loss:0.20641877, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33916 sec, avg_samples: 4096.00000, ips: 12076.80625 ins/s2024-05-11 19:55:59,994 - INFO - epoch: 0, batch_id: 800, auc:0.600670, loss:0.22155708, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33848 sec, avg_samples: 4096.00000, ips: 12101.22339 ins/s2024-05-11 19:56:17,091 - INFO - epoch: 0, batch_id: 850, auc:0.603358, loss:0.19764367, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34137 sec, avg_samples: 4096.00000, ips: 11998.85636 ins/s2024-05-11 19:56:34,397 - INFO - epoch: 0, batch_id: 900, auc:0.605445, loss:0.18218887, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.34556 sec, avg_samples: 4096.00000, ips: 11853.31707 ins/s2024-05-11 19:56:53,374 - INFO - epoch: 0, batch_id: 950, auc:0.606719, loss:0.20349224, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.37895 sec, avg_samples: 4096.00000, ips: 10808.89367 ins/s2024-05-11 19:57:12,244 - INFO - epoch: 0, batch_id: 1000, auc:0.608219, loss:0.18338634, avg_reader_cost: 0.00016 sec, avg_batch_cost: 0.37685 sec, avg_samples: 4096.00000, ips: 10868.97179 ins/s2024-05-11 19:57:30,490 - INFO - epoch: 0, batch_id: 1050, auc:0.610018, loss:0.18991007, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.36437 sec, avg_samples: 4096.00000, ips: 11241.38734 ins/s2024-05-11 19:57:48,290 - INFO - epoch: 0, batch_id: 1100, auc:0.611764, loss:0.19425409, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.35542 sec, avg_samples: 4096.00000, ips: 11524.47769 ins/s2024-05-11 19:58:05,738 - INFO - epoch: 0, batch_id: 1150, auc:0.613360, loss:0.18417387, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34839 sec, avg_samples: 4096.00000, ips: 11756.97841 ins/s2024-05-11 19:58:22,780 - INFO - epoch: 0, batch_id: 1200, auc:0.615447, loss:0.2374034, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34027 sec, avg_samples: 4096.00000, ips: 12037.41497 ins/s2024-05-11 19:58:39,730 - INFO - epoch: 0, batch_id: 1250, auc:0.616718, loss:0.21474466, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.33845 sec, avg_samples: 4096.00000, ips: 12102.39913 ins/s2024-05-11 19:58:56,387 - INFO - epoch: 0, batch_id: 1300, auc:0.618325, loss:0.17899244, avg_reader_cost: 0.00016 sec, avg_batch_cost: 0.33259 sec, avg_samples: 4096.00000, ips: 12315.36361 ins/s2024-05-11 19:59:13,529 - INFO - epoch: 0, batch_id: 1350, auc:0.619961, loss:0.21630415, avg_reader_cost: 0.00015 sec, avg_batch_cost: 0.34231 sec, avg_samples: 4096.00000, ips: 11965.62220 ins/s2024-05-11 19:59:14,210 - INFO - epoch: 0 done, auc: 0.620026,loss:0.14849854, epoch time: 480.97 s2024-05-11 19:59:14,386 - INFO - Already save model in output_model_all_dsin/0

登录后复制In [10]

!python ../../../tools/infer.py -m config_bigdata.yaml

登录后复制

2024-05-11 19:59:48,026 - INFO - **************common.configs**********2024-05-11 19:59:48,026 - INFO - use_gpu: True, use_xpu: False, use_visual: False, infer_batch_size: 16384, test_data_dir: ../../../datasets/Ali_Display_Ad_Click_DSIN/big_test/, start_epoch: 0, end_epoch: 1, print_interval: 50, model_load_path: output_model_all_dsin2024-05-11 19:59:48,026 - INFO - **************common.configs**********W0511 19:59:48.027812  1904 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1W0511 19:59:48.033318  1904 device_context.cc:465] device: 0, cuDNN Version: 7.6.2024-05-11 19:59:52,275 - INFO - read data2024-05-11 19:59:52,276 - INFO - reader path:dsin_reader2024-05-11 19:59:53,777 - INFO - load model epoch 02024-05-11 19:59:53,777 - INFO - start load model from output_model_all_dsin/02024-05-11 19:59:54,438 - INFO - epoch: 0, batch_id: 0, auc: 0.628742, avg_reader_cost: 0.00439 sec, avg_batch_cost: 0.01166 sec, avg_samples: 16384.00000, ips: 1239157.77 ins/s2024-05-11 20:00:02,133 - INFO - epoch: 0 done, auc: 0.635660, epoch time: 8.36 s

登录后复制

六、代码结构与详细说明

6.1 代码结构(在work/PaddleRec/models/rank/dsin目录下)

├── config_bigdata.yaml # 全量数据配置文件├── net.py # 模型核心组网（动静统一）├── sequence_layers.py # 模型组网模块├── dsin_reader.py # 数据读取程序├── dygraph_model.py # 构建动态图

登录后复制

6.2 参数说明

可以在 config_bigdata.yaml 中设置训练与评估相关参数，具体如下：

评估的相关参数与训练类似就不展开了，且模型的超参数也详细存放在了config_bigdata.yaml文件中了。

七、复现心得

本项目复现时遇到了许多的问题，分别对应在数据集获取，模型对齐，精度对齐这几方面。且主要因为源代码模块化程度较高，把很多细节都需要去仔细阅读源码才能彻底明白模型框架，论文上的模型框架图只是提供一个大致的思路。

(1)数据集获取： 因为该论文所使用的数据集是需要经过预处理的，而且原始数据集大小为23G+，所以在一开始复现的时候就在数据集这上面费了不少心思，也正是因为这样我学习到了如何去处理这种大数据。通过逐行读取，划分成多个子数据集的方式，将用户的历史行为数据切分成了多个。

(2)模型对齐： 如前面所说的，在一开始复现的时候我就只通过原文中的模型框架尝试去搭建，可想而知结果肯定是不好的。还是需要去仔细阅读原文代码才能很好的与原文模型进行对齐。(可以通过model summary的方式去观察模型框架以及每个layer的输入输出shape。)

(3)精度对齐： 在精度对齐上面，实在是太费心思了。重点是确保数据集一定要与原文一致，模型一定要与原文对齐。达到这两点后基本上就可以实现精度对齐了。

八、模型信息

训练完成后，模型和相关LOG保存在./output_model_all_dsin目录下。

In [ ]

登录后复制